2024-03-29T11:07:01.296 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T11:07:01.296 DEBUG:teuthology.run:Teuthology command: teuthology --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} --owner scheduled_amathuri@teuthology --verbose --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622683 -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622683/orig.config.yaml 2024-03-29T11:07:01.335 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:07:01.428 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622683 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622683' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi017.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGZcCOJjiFUA2a6jFFSoq93F3qAol6z1qPchs7Cw7fNOZHqYW7tnklnfaMxeM5zo+PIdJdeOTOw9XjhdIP5sXs0= smithi094.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFplVfDj+ocYFtAOZXcnsLnocc/g2p4HL0RJGa2ZhOlaiDHJzgpAQBJR8Rqbd8sQotCrDibgV/AXAR1qM2gc4VI= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T11:07:01.429 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T11:07:01.429 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T11:07:01.430 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T11:07:01.431 INFO:teuthology.task.internal:Checking packages... 2024-03-29T11:07:01.456 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T11:07:01.457 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T11:07:01.457 INFO:teuthology.packaging:ref: None 2024-03-29T11:07:01.457 INFO:teuthology.packaging:tag: None 2024-03-29T11:07:01.457 INFO:teuthology.packaging:branch: main 2024-03-29T11:07:01.457 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:07:01.457 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T11:07:01.834 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T11:07:01.836 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T11:07:01.844 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T11:07:01.845 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T11:07:01.871 INFO:teuthology.task.internal:Saving configuration 2024-03-29T11:07:01.885 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T11:07:01.901 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:07:01.928 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi017.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622683', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:02:04.965790', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGZcCOJjiFUA2a6jFFSoq93F3qAol6z1qPchs7Cw7fNOZHqYW7tnklnfaMxeM5zo+PIdJdeOTOw9XjhdIP5sXs0='} 2024-03-29T11:07:01.954 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi094.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622683', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:02:04.967455', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFplVfDj+ocYFtAOZXcnsLnocc/g2p4HL0RJGa2ZhOlaiDHJzgpAQBJR8Rqbd8sQotCrDibgV/AXAR1qM2gc4VI='} 2024-03-29T11:07:01.954 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T11:07:01.966 INFO:teuthology.task.internal:roles: ubuntu@smithi017.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T11:07:01.966 INFO:teuthology.task.internal:roles: ubuntu@smithi094.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T11:07:01.966 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T11:07:02.057 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f28551530d0>, signals=[15]) 2024-03-29T11:07:02.057 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T11:07:02.066 INFO:teuthology.task.internal:Opening connections... 2024-03-29T11:07:02.066 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi017.front.sepia.ceph.com 2024-03-29T11:07:02.068 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:07:02.176 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi094.front.sepia.ceph.com 2024-03-29T11:07:02.177 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:07:02.265 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T11:07:02.278 DEBUG:teuthology.orchestra.run.smithi017:> uname -m 2024-03-29T11:07:02.296 INFO:teuthology.orchestra.run.smithi017.stdout:x86_64 2024-03-29T11:07:02.296 DEBUG:teuthology.orchestra.run.smithi017:> cat /etc/os-release 2024-03-29T11:07:02.354 INFO:teuthology.orchestra.run.smithi017.stdout:NAME="CentOS Stream" 2024-03-29T11:07:02.354 INFO:teuthology.orchestra.run.smithi017.stdout:VERSION="9" 2024-03-29T11:07:02.354 INFO:teuthology.orchestra.run.smithi017.stdout:ID="centos" 2024-03-29T11:07:02.354 INFO:teuthology.orchestra.run.smithi017.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:07:02.354 INFO:teuthology.orchestra.run.smithi017.stdout:VERSION_ID="9" 2024-03-29T11:07:02.354 INFO:teuthology.orchestra.run.smithi017.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:07:02.354 INFO:teuthology.orchestra.run.smithi017.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:07:02.355 INFO:teuthology.orchestra.run.smithi017.stdout:ANSI_COLOR="0;31" 2024-03-29T11:07:02.355 INFO:teuthology.orchestra.run.smithi017.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:07:02.355 INFO:teuthology.orchestra.run.smithi017.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:07:02.355 INFO:teuthology.orchestra.run.smithi017.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:07:02.355 INFO:teuthology.orchestra.run.smithi017.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:07:02.355 INFO:teuthology.orchestra.run.smithi017.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:07:02.355 INFO:teuthology.orchestra.run.smithi017.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:07:02.356 INFO:teuthology.lock.ops:Updating smithi017.front.sepia.ceph.com on lock server 2024-03-29T11:07:02.377 DEBUG:teuthology.orchestra.run.smithi094:> uname -m 2024-03-29T11:07:02.395 INFO:teuthology.orchestra.run.smithi094.stdout:x86_64 2024-03-29T11:07:02.395 DEBUG:teuthology.orchestra.run.smithi094:> cat /etc/os-release 2024-03-29T11:07:02.452 INFO:teuthology.orchestra.run.smithi094.stdout:NAME="CentOS Stream" 2024-03-29T11:07:02.452 INFO:teuthology.orchestra.run.smithi094.stdout:VERSION="9" 2024-03-29T11:07:02.452 INFO:teuthology.orchestra.run.smithi094.stdout:ID="centos" 2024-03-29T11:07:02.452 INFO:teuthology.orchestra.run.smithi094.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:07:02.453 INFO:teuthology.orchestra.run.smithi094.stdout:VERSION_ID="9" 2024-03-29T11:07:02.453 INFO:teuthology.orchestra.run.smithi094.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:07:02.453 INFO:teuthology.orchestra.run.smithi094.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:07:02.453 INFO:teuthology.orchestra.run.smithi094.stdout:ANSI_COLOR="0;31" 2024-03-29T11:07:02.453 INFO:teuthology.orchestra.run.smithi094.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:07:02.453 INFO:teuthology.orchestra.run.smithi094.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:07:02.453 INFO:teuthology.orchestra.run.smithi094.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:07:02.453 INFO:teuthology.orchestra.run.smithi094.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:07:02.453 INFO:teuthology.orchestra.run.smithi094.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:07:02.453 INFO:teuthology.orchestra.run.smithi094.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:07:02.453 INFO:teuthology.lock.ops:Updating smithi094.front.sepia.ceph.com on lock server 2024-03-29T11:07:02.477 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T11:07:02.488 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T11:07:02.506 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T11:07:02.507 DEBUG:teuthology.orchestra.run.smithi017:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:07:02.509 DEBUG:teuthology.orchestra.run.smithi094:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:07:02.525 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T11:07:02.535 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T11:07:02.535 DEBUG:teuthology.orchestra.run.smithi017:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:07:02.566 DEBUG:teuthology.orchestra.run.smithi094:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:07:02.650 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T11:07:02.706 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T11:07:02.725 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T11:07:02.725 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T11:07:02.726 DEBUG:teuthology.orchestra.run.smithi017:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:07:02.727 DEBUG:teuthology.orchestra.run.smithi094:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:07:02.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:07:02.743 DEBUG:teuthology.orchestra.run.smithi094:> uname -r 2024-03-29T11:07:02.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:07:02.744 DEBUG:teuthology.orchestra.run.smithi017:> uname -r 2024-03-29T11:07:02.801 INFO:teuthology.orchestra.run.smithi094.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:07:02.802 INFO:teuthology.task.kernel:Running kernel on smithi094: 5.14.0-361.el9.x86_64 2024-03-29T11:07:02.802 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum install -y kernel 2024-03-29T11:07:02.803 INFO:teuthology.orchestra.run.smithi017.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:07:02.803 INFO:teuthology.task.kernel:Running kernel on smithi017: 5.14.0-361.el9.x86_64 2024-03-29T11:07:02.804 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum install -y kernel 2024-03-29T11:07:04.895 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-29T11:07:05.117 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-29T11:07:06.353 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS-9 - BaseOS 6.5 MB/s | 2.0 MB 00:00 2024-03-29T11:07:06.534 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS-9 - BaseOS 6.8 MB/s | 2.0 MB 00:00 2024-03-29T11:07:07.942 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - BaseOS 7.5 MB/s | 8.0 MB 00:01 2024-03-29T11:07:10.085 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - BaseOS 2.4 MB/s | 8.0 MB 00:03 2024-03-29T11:07:15.300 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - AppStream 3.0 MB/s | 19 MB 00:06 2024-03-29T11:07:19.485 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - CRB 9.1 MB/s | 6.1 MB 00:00 2024-03-29T11:07:21.095 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 16 kB 00:00 2024-03-29T11:07:25.693 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - AppStream 1.3 MB/s | 19 MB 00:14 2024-03-29T11:07:29.802 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-29T11:07:32.152 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - Extras packages 13 kB/s | 16 kB 00:01 2024-03-29T11:07:34.745 INFO:teuthology.orchestra.run.smithi017.stdout:Extra Packages for Enterprise Linux 8.4 MB/s | 21 MB 00:02 2024-03-29T11:07:40.004 INFO:teuthology.orchestra.run.smithi017.stdout:lab-extras 43 kB/s | 1.7 kB 00:00 2024-03-29T11:07:42.369 INFO:teuthology.orchestra.run.smithi017.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:07:42.369 INFO:teuthology.orchestra.run.smithi017.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:07:42.464 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-03-29T11:07:42.470 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-03-29T11:07:42.470 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repository Size 2024-03-29T11:07:42.470 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-03-29T11:07:42.470 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-03-29T11:07:42.470 INFO:teuthology.orchestra.run.smithi017.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:07:42.470 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2024-03-29T11:07:42.470 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:07:42.470 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:07:42.470 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:07:42.470 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:07:42.470 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-03-29T11:07:42.471 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-03-29T11:07:42.471 INFO:teuthology.orchestra.run.smithi017.stdout:Install 4 Packages 2024-03-29T11:07:42.471 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:07:42.471 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 76 M 2024-03-29T11:07:42.471 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 123 M 2024-03-29T11:07:42.471 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-03-29T11:07:42.613 INFO:teuthology.orchestra.run.smithi017.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-03-29T11:07:43.555 INFO:teuthology.orchestra.run.smithi017.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 14 MB/s | 15 MB 00:01 2024-03-29T11:07:44.614 INFO:teuthology.orchestra.run.smithi017.stdout:(3/4): kernel-modules-core-5.14.0-434.el9.x86_6 14 MB/s | 28 MB 00:01 2024-03-29T11:07:45.056 INFO:teuthology.orchestra.run.smithi017.stdout:(4/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 13 MB/s | 34 MB 00:02 2024-03-29T11:07:45.062 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:07:45.062 INFO:teuthology.orchestra.run.smithi017.stdout:Total 30 MB/s | 76 MB 00:02 2024-03-29T11:07:45.063 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-03-29T11:07:45.472 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-03-29T11:07:45.472 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-03-29T11:07:46.358 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-03-29T11:07:46.359 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-03-29T11:07:48.783 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-03-29T11:07:49.403 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:07:49.489 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:07:52.384 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:07:52.516 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:07:54.835 INFO:teuthology.orchestra.run.smithi094.stdout:Extra Packages for Enterprise Linux 632 kB/s | 21 MB 00:33 2024-03-29T11:07:57.585 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:07:57.624 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:07:59.785 INFO:teuthology.orchestra.run.smithi094.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-03-29T11:08:01.992 INFO:teuthology.orchestra.run.smithi094.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:08:01.992 INFO:teuthology.orchestra.run.smithi094.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:08:02.090 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T11:08:02.101 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T11:08:02.101 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-03-29T11:08:02.101 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T11:08:02.101 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-29T11:08:02.101 INFO:teuthology.orchestra.run.smithi094.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:08:02.101 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-03-29T11:08:02.101 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:08:02.101 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:08:02.101 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:08:02.101 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:08:02.102 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-29T11:08:02.102 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T11:08:02.102 INFO:teuthology.orchestra.run.smithi094.stdout:Install 4 Packages 2024-03-29T11:08:02.102 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:08:02.102 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 76 M 2024-03-29T11:08:02.102 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 123 M 2024-03-29T11:08:02.102 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-29T11:08:02.253 INFO:teuthology.orchestra.run.smithi094.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-29T11:08:02.717 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:02.828 INFO:teuthology.orchestra.run.smithi094.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 21 MB/s | 15 MB 00:00 2024-03-29T11:08:03.454 INFO:teuthology.orchestra.run.smithi094.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 25 MB/s | 34 MB 00:01 2024-03-29T11:08:03.729 INFO:teuthology.orchestra.run.smithi094.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 19 MB/s | 28 MB 00:01 2024-03-29T11:08:03.732 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:08:03.732 INFO:teuthology.orchestra.run.smithi094.stdout:Total 47 MB/s | 76 MB 00:01 2024-03-29T11:08:03.732 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-29T11:08:04.102 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-29T11:08:04.102 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-29T11:08:04.963 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-29T11:08:04.963 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-29T11:08:07.269 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-29T11:08:07.952 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:08:08.043 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:08:10.822 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:08:10.959 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:08:15.705 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:08:15.742 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:20.478 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:33.480 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:33.480 INFO:teuthology.orchestra.run.smithi017.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:08:33.480 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:08:33.525 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:33.851 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:33.851 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:08:33.851 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:08:33.852 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:08:34.294 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:34.294 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:08:34.294 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-03-29T11:08:34.294 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:08:34.295 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:08:34.295 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:08:34.295 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:08:34.295 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:08:34.295 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-03-29T11:08:34.616 DEBUG:teuthology.orchestra.run.smithi017:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:08:34.717 INFO:teuthology.orchestra.run.smithi017.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:08:34.718 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:08:34.718 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:08:34.718 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:08:34.718 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:08:34.718 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T11:08:34.718 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi017.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:08:34.719 DEBUG:teuthology.orchestra.run.smithi017:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:08:34.801 INFO:teuthology.orchestra.run.smithi017.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:08:34.801 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:08:34.801 DEBUG:teuthology.orchestra.run.smithi017:> sudo rpm -qi grub2-tools 2024-03-29T11:08:34.838 INFO:teuthology.orchestra.run.smithi017.stdout:Name : grub2-tools 2024-03-29T11:08:34.839 INFO:teuthology.orchestra.run.smithi017.stdout:Epoch : 1 2024-03-29T11:08:34.839 INFO:teuthology.orchestra.run.smithi017.stdout:Version : 2.06 2024-03-29T11:08:34.839 INFO:teuthology.orchestra.run.smithi017.stdout:Release : 68.el9 2024-03-29T11:08:34.839 INFO:teuthology.orchestra.run.smithi017.stdout:Architecture: x86_64 2024-03-29T11:08:34.839 INFO:teuthology.orchestra.run.smithi017.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:08:34.839 INFO:teuthology.orchestra.run.smithi017.stdout:Group : Unspecified 2024-03-29T11:08:34.839 INFO:teuthology.orchestra.run.smithi017.stdout:Size : 8264254 2024-03-29T11:08:34.839 INFO:teuthology.orchestra.run.smithi017.stdout:License : GPLv3+ 2024-03-29T11:08:34.839 INFO:teuthology.orchestra.run.smithi017.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:08:34.839 INFO:teuthology.orchestra.run.smithi017.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:08:34.839 INFO:teuthology.orchestra.run.smithi017.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:08:34.839 INFO:teuthology.orchestra.run.smithi017.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:08:34.839 INFO:teuthology.orchestra.run.smithi017.stdout:Packager : builder@centos.org 2024-03-29T11:08:34.839 INFO:teuthology.orchestra.run.smithi017.stdout:Vendor : CentOS 2024-03-29T11:08:34.840 INFO:teuthology.orchestra.run.smithi017.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:08:34.840 INFO:teuthology.orchestra.run.smithi017.stdout:Summary : Support tools for GRUB. 2024-03-29T11:08:34.840 INFO:teuthology.orchestra.run.smithi017.stdout:Description : 2024-03-29T11:08:34.840 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:08:34.840 INFO:teuthology.orchestra.run.smithi017.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:08:34.840 INFO:teuthology.orchestra.run.smithi017.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:08:34.840 INFO:teuthology.orchestra.run.smithi017.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:08:34.840 INFO:teuthology.orchestra.run.smithi017.stdout:hardware devices. 2024-03-29T11:08:34.840 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:08:34.840 INFO:teuthology.orchestra.run.smithi017.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:08:34.841 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:08:34.841 INFO:teuthology.task.kernel:Updating grub on smithi017 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:08:34.841 DEBUG:teuthology.orchestra.run.smithi017:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:08:35.512 INFO:teuthology.orchestra.run.smithi017.stderr:Generating grub configuration file ... 2024-03-29T11:08:36.825 INFO:teuthology.orchestra.run.smithi017.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:08:36.853 INFO:teuthology.orchestra.run.smithi017.stderr:done 2024-03-29T11:08:36.855 DEBUG:teuthology.orchestra.run.smithi017:> mktemp 2024-03-29T11:08:36.870 INFO:teuthology.orchestra.run.smithi017.stdout:/tmp/tmp.xnLJc9SeNz 2024-03-29T11:08:36.871 DEBUG:teuthology.orchestra.run.smithi017:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.xnLJc9SeNz 2024-03-29T11:08:36.942 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0666 /tmp/tmp.xnLJc9SeNz 2024-03-29T11:08:37.089 DEBUG:teuthology.orchestra.remote:smithi017:/tmp/tmp.xnLJc9SeNz is 6KB 2024-03-29T11:08:37.137 DEBUG:teuthology.orchestra.run.smithi017:> rm -fr /tmp/tmp.xnLJc9SeNz 2024-03-29T11:08:37.152 DEBUG:teuthology.orchestra.run.smithi017:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:08:37.230 INFO:teuthology.orchestra.run.smithi017.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:08:37.230 INFO:teuthology.orchestra.run.smithi017.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:08:37.230 INFO:teuthology.orchestra.run.smithi017.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:08:37.230 INFO:teuthology.orchestra.run.smithi017.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:08:37.232 DEBUG:teuthology.orchestra.run.smithi017:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:08:37.380 DEBUG:teuthology.orchestra.run.smithi017:> sudo shutdown -r now 2024-03-29T11:08:50.073 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:50.073 INFO:teuthology.orchestra.run.smithi094.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:08:50.073 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:08:50.116 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:50.504 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:50.504 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:08:50.504 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:08:50.504 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:08:50.942 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:50.942 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:08:50.943 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-29T11:08:50.943 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:08:50.943 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:08:50.943 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:08:50.943 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:08:50.943 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:08:50.943 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T11:08:51.277 DEBUG:teuthology.orchestra.run.smithi094:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:08:51.376 INFO:teuthology.orchestra.run.smithi094.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:08:51.376 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:08:51.376 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:08:51.376 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:08:51.376 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:08:51.377 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T11:08:51.377 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi094.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:08:51.377 DEBUG:teuthology.orchestra.run.smithi094:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:08:51.457 INFO:teuthology.orchestra.run.smithi094.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:08:51.457 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:08:51.457 DEBUG:teuthology.orchestra.run.smithi094:> sudo rpm -qi grub2-tools 2024-03-29T11:08:51.497 INFO:teuthology.orchestra.run.smithi094.stdout:Name : grub2-tools 2024-03-29T11:08:51.497 INFO:teuthology.orchestra.run.smithi094.stdout:Epoch : 1 2024-03-29T11:08:51.497 INFO:teuthology.orchestra.run.smithi094.stdout:Version : 2.06 2024-03-29T11:08:51.497 INFO:teuthology.orchestra.run.smithi094.stdout:Release : 68.el9 2024-03-29T11:08:51.498 INFO:teuthology.orchestra.run.smithi094.stdout:Architecture: x86_64 2024-03-29T11:08:51.498 INFO:teuthology.orchestra.run.smithi094.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:08:51.498 INFO:teuthology.orchestra.run.smithi094.stdout:Group : Unspecified 2024-03-29T11:08:51.498 INFO:teuthology.orchestra.run.smithi094.stdout:Size : 8264254 2024-03-29T11:08:51.498 INFO:teuthology.orchestra.run.smithi094.stdout:License : GPLv3+ 2024-03-29T11:08:51.498 INFO:teuthology.orchestra.run.smithi094.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:08:51.498 INFO:teuthology.orchestra.run.smithi094.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:08:51.498 INFO:teuthology.orchestra.run.smithi094.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:08:51.498 INFO:teuthology.orchestra.run.smithi094.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:08:51.498 INFO:teuthology.orchestra.run.smithi094.stdout:Packager : builder@centos.org 2024-03-29T11:08:51.498 INFO:teuthology.orchestra.run.smithi094.stdout:Vendor : CentOS 2024-03-29T11:08:51.498 INFO:teuthology.orchestra.run.smithi094.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:08:51.498 INFO:teuthology.orchestra.run.smithi094.stdout:Summary : Support tools for GRUB. 2024-03-29T11:08:51.498 INFO:teuthology.orchestra.run.smithi094.stdout:Description : 2024-03-29T11:08:51.499 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:08:51.499 INFO:teuthology.orchestra.run.smithi094.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:08:51.499 INFO:teuthology.orchestra.run.smithi094.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:08:51.499 INFO:teuthology.orchestra.run.smithi094.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:08:51.499 INFO:teuthology.orchestra.run.smithi094.stdout:hardware devices. 2024-03-29T11:08:51.499 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:08:51.499 INFO:teuthology.orchestra.run.smithi094.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:08:51.500 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:08:51.500 INFO:teuthology.task.kernel:Updating grub on smithi094 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:08:51.500 DEBUG:teuthology.orchestra.run.smithi094:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:08:52.191 INFO:teuthology.orchestra.run.smithi094.stderr:Generating grub configuration file ... 2024-03-29T11:08:53.584 INFO:teuthology.orchestra.run.smithi094.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:08:53.633 INFO:teuthology.orchestra.run.smithi094.stderr:done 2024-03-29T11:08:53.636 DEBUG:teuthology.orchestra.run.smithi094:> mktemp 2024-03-29T11:08:53.651 INFO:teuthology.orchestra.run.smithi094.stdout:/tmp/tmp.95SlrzHCY6 2024-03-29T11:08:53.651 DEBUG:teuthology.orchestra.run.smithi094:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.95SlrzHCY6 2024-03-29T11:08:53.720 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0666 /tmp/tmp.95SlrzHCY6 2024-03-29T11:08:53.861 DEBUG:teuthology.orchestra.remote:smithi094:/tmp/tmp.95SlrzHCY6 is 6KB 2024-03-29T11:08:53.881 DEBUG:teuthology.orchestra.run.smithi094:> rm -fr /tmp/tmp.95SlrzHCY6 2024-03-29T11:08:53.894 DEBUG:teuthology.orchestra.run.smithi094:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:08:53.959 INFO:teuthology.orchestra.run.smithi094.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:08:53.959 INFO:teuthology.orchestra.run.smithi094.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:08:53.959 INFO:teuthology.orchestra.run.smithi094.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:08:53.959 INFO:teuthology.orchestra.run.smithi094.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:08:53.960 DEBUG:teuthology.orchestra.run.smithi094:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:08:54.124 DEBUG:teuthology.orchestra.run.smithi094:> sudo shutdown -r now 2024-03-29T11:09:07.389 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T11:09:07.389 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-03-29T11:09:07.390 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:09:24.129 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T11:09:24.129 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-03-29T11:09:24.130 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:09:25.918 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.17 2024-03-29T11:09:34.928 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-03-29T11:09:34.929 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:09:42.558 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.94 2024-03-29T11:09:51.568 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-03-29T11:09:51.569 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:09:54.622 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.94 2024-03-29T11:10:06.626 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-03-29T11:10:06.627 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:10:34.958 DEBUG:teuthology.orchestra.remote:timed out 2024-03-29T11:10:38.439 DEBUG:teuthology.orchestra.run.smithi094:> true 2024-03-29T11:10:38.881 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-03-29T11:10:38.881 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:10:38.882 DEBUG:teuthology.orchestra.run.smithi094:> uname -r 2024-03-29T11:10:38.895 INFO:teuthology.orchestra.run.smithi094.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:10:38.896 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:10:38.896 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:10:38.896 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:10:39.897 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T11:10:39.897 DEBUG:teuthology.orchestra.run.smithi094:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:10:39.979 INFO:teuthology.orchestra.run.smithi094.stdout:ttyS1 2024-03-29T11:10:40.030 DEBUG:teuthology.parallel:result is None 2024-03-29T11:10:46.959 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-03-29T11:10:46.960 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:10:47.346 DEBUG:teuthology.orchestra.run.smithi017:> true 2024-03-29T11:10:47.727 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-03-29T11:10:47.728 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:10:47.728 DEBUG:teuthology.orchestra.run.smithi017:> uname -r 2024-03-29T11:10:47.744 INFO:teuthology.orchestra.run.smithi017.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:10:47.744 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:10:47.744 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:10:47.744 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:10:48.746 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T11:10:48.746 DEBUG:teuthology.orchestra.run.smithi017:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:10:48.829 INFO:teuthology.orchestra.run.smithi017.stdout:ttyS1 2024-03-29T11:10:48.857 DEBUG:teuthology.parallel:result is None 2024-03-29T11:10:48.857 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T11:10:48.872 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T11:10:48.872 DEBUG:teuthology.orchestra.run.smithi017:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:10:48.875 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:10:48.893 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T11:10:48.905 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T11:10:48.936 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T11:10:48.936 DEBUG:teuthology.orchestra.run.smithi017:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:10:48.938 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:10:48.990 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T11:10:49.002 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T11:10:49.002 DEBUG:teuthology.orchestra.run.smithi017:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:10:49.005 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:10:49.037 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:10:49.041 INFO:teuthology.orchestra.run.smithi017.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:10:49.051 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:10:49.056 INFO:teuthology.orchestra.run.smithi017.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:10:49.058 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T11:10:49.069 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T11:10:49.069 DEBUG:teuthology.orchestra.run.smithi017:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:10:49.101 DEBUG:teuthology.orchestra.run.smithi094:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:10:49.130 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T11:10:49.144 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T11:10:49.144 DEBUG:teuthology.orchestra.run.smithi017:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:10:49.172 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:10:49.187 DEBUG:teuthology.orchestra.run.smithi017:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:10:49.263 DEBUG:teuthology.orchestra.run.smithi017:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:10:49.325 DEBUG:teuthology.orchestra.run.smithi017:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:10:49.405 DEBUG:teuthology.orchestra.run.smithi017:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:10:49.453 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-03-29T11:10:49.453 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:10:49.522 DEBUG:teuthology.orchestra.run.smithi094:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:10:49.561 DEBUG:teuthology.orchestra.run.smithi094:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:10:49.618 DEBUG:teuthology.orchestra.run.smithi094:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:10:49.701 DEBUG:teuthology.orchestra.run.smithi094:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:10:49.746 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-29T11:10:49.746 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:10:49.815 DEBUG:teuthology.orchestra.run.smithi017:> sudo service rsyslog restart 2024-03-29T11:10:49.818 DEBUG:teuthology.orchestra.run.smithi094:> sudo service rsyslog restart 2024-03-29T11:10:49.864 INFO:teuthology.orchestra.run.smithi017.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:10:49.902 INFO:teuthology.orchestra.run.smithi094.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:10:50.316 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T11:10:50.330 INFO:teuthology.task.internal:Starting timer... 2024-03-29T11:10:50.330 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T11:10:50.362 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T11:10:50.396 DEBUG:teuthology.orchestra.run.smithi017:> sudo service auditd rotate 2024-03-29T11:10:50.455 INFO:teuthology.orchestra.run.smithi017.stdout:Rotating logs: 2024-03-29T11:10:50.457 DEBUG:teuthology.orchestra.run.smithi094:> sudo service auditd rotate 2024-03-29T11:10:50.520 INFO:teuthology.orchestra.run.smithi094.stdout:Rotating logs: 2024-03-29T11:10:50.521 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T11:10:50.522 DEBUG:teuthology.orchestra.run.smithi017:> /usr/sbin/getenforce 2024-03-29T11:10:50.540 INFO:teuthology.orchestra.run.smithi017.stdout:Permissive 2024-03-29T11:10:50.541 DEBUG:teuthology.orchestra.run.smithi094:> /usr/sbin/getenforce 2024-03-29T11:10:50.571 INFO:teuthology.orchestra.run.smithi094.stdout:Permissive 2024-03-29T11:10:50.572 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi017.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi094.front.sepia.ceph.com': 'permissive'} 2024-03-29T11:10:50.572 DEBUG:teuthology.orchestra.run.smithi017:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:10:50.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:10:50.610 DEBUG:teuthology.orchestra.run.smithi094:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:10:50.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:10:50.641 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T11:10:50.641 DEBUG:teuthology.orchestra.run.smithi017:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:10:50.684 DEBUG:teuthology.orchestra.run.smithi094:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:10:50.715 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T11:10:50.727 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T11:10:50.744 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T11:10:50.746 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi017.front.sepia.ceph.com,smithi094.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T11:15:09.802 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi017.front.sepia.ceph.com'), Remote(name='ubuntu@smithi094.front.sepia.ceph.com')] 2024-03-29T11:15:09.803 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-03-29T11:15:09.804 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:15:09.899 DEBUG:teuthology.orchestra.run.smithi017:> true 2024-03-29T11:15:09.982 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-03-29T11:15:09.982 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-03-29T11:15:09.983 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:15:10.072 DEBUG:teuthology.orchestra.run.smithi094:> true 2024-03-29T11:15:10.162 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-03-29T11:15:10.162 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T11:15:10.172 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T11:15:10.172 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:15:10.173 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:15:10.175 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:15:10.175 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:15:10.205 INFO:teuthology.orchestra.run.smithi017.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:15:10.222 INFO:teuthology.orchestra.run.smithi017.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:15:10.234 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:15:10.251 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:15:10.252 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: ntpd: command not found 2024-03-29T11:15:10.266 INFO:teuthology.orchestra.run.smithi017.stdout:506 Cannot talk to daemon 2024-03-29T11:15:10.281 INFO:teuthology.orchestra.run.smithi094.stderr:sudo: ntpd: command not found 2024-03-29T11:15:10.282 INFO:teuthology.orchestra.run.smithi017.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:15:10.294 INFO:teuthology.orchestra.run.smithi094.stdout:506 Cannot talk to daemon 2024-03-29T11:15:10.299 INFO:teuthology.orchestra.run.smithi017.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:15:10.311 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:15:10.327 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:15:10.350 INFO:teuthology.orchestra.run.smithi017.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:15:10.354 INFO:teuthology.orchestra.run.smithi017.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:15:10.354 INFO:teuthology.orchestra.run.smithi017.stdout:=============================================================================== 2024-03-29T11:15:10.354 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:10.354 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:10.354 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:10.354 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:10.380 INFO:teuthology.orchestra.run.smithi094.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:15:10.384 INFO:teuthology.orchestra.run.smithi094.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:15:10.384 INFO:teuthology.orchestra.run.smithi094.stdout:=============================================================================== 2024-03-29T11:15:10.384 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:10.384 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:10.384 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:10.384 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:10.384 INFO:teuthology.run_tasks:Running task install... 2024-03-29T11:15:10.396 DEBUG:teuthology.task.install:project ceph 2024-03-29T11:15:10.396 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T11:15:10.396 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T11:15:10.396 INFO:teuthology.task.install:Using flavor: default 2024-03-29T11:15:10.407 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T11:15:10.408 INFO:teuthology.task.install:extra packages: [] 2024-03-29T11:15:10.408 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:15:10.409 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:15:10.412 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:15:10.412 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:15:10.568 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:15:10.568 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:15:10.572 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:15:10.572 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:15:10.679 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:15:10.680 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-03-29T11:15:10.680 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:15:10.686 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:15:10.686 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-29T11:15:10.686 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:15:10.708 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:15:10.708 DEBUG:teuthology.orchestra.run.smithi017:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:15:10.714 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:15:10.714 DEBUG:teuthology.orchestra.run.smithi094:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:15:10.777 DEBUG:teuthology.orchestra.run.smithi017:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:15:10.782 DEBUG:teuthology.orchestra.run.smithi094:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:15:10.857 DEBUG:teuthology.orchestra.run.smithi017:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:15:10.862 DEBUG:teuthology.orchestra.run.smithi094:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:15:10.924 INFO:teuthology.orchestra.run.smithi017.stdout:check_obsoletes = 1 2024-03-29T11:15:10.925 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum clean all 2024-03-29T11:15:10.928 INFO:teuthology.orchestra.run.smithi094.stdout:check_obsoletes = 1 2024-03-29T11:15:10.930 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum clean all 2024-03-29T11:15:11.172 INFO:teuthology.orchestra.run.smithi017.stdout:74 files removed 2024-03-29T11:15:11.179 INFO:teuthology.orchestra.run.smithi094.stdout:74 files removed 2024-03-29T11:15:11.199 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-radosgw 2024-03-29T11:15:11.205 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-radosgw 2024-03-29T11:15:11.890 INFO:teuthology.orchestra.run.smithi017.stdout:ceph packages for x86_64 213 kB/s | 82 kB 00:00 2024-03-29T11:15:11.946 INFO:teuthology.orchestra.run.smithi094.stdout:ceph packages for x86_64 185 kB/s | 82 kB 00:00 2024-03-29T11:15:12.315 INFO:teuthology.orchestra.run.smithi017.stdout:ceph noarch packages 27 kB/s | 10 kB 00:00 2024-03-29T11:15:12.389 INFO:teuthology.orchestra.run.smithi094.stdout:ceph noarch packages 25 kB/s | 10 kB 00:00 2024-03-29T11:15:12.714 INFO:teuthology.orchestra.run.smithi094.stdout:ceph source packages 6.6 kB/s | 1.9 kB 00:00 2024-03-29T11:15:12.724 INFO:teuthology.orchestra.run.smithi017.stdout:ceph source packages 5.2 kB/s | 1.9 kB 00:00 2024-03-29T11:15:13.142 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-29T11:15:13.235 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-29T11:15:14.685 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS-9 - BaseOS 7.1 MB/s | 2.0 MB 00:00 2024-03-29T11:15:14.758 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS-9 - BaseOS 4.1 MB/s | 2.0 MB 00:00 2024-03-29T11:15:15.883 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-03-29T11:15:16.610 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - BaseOS 5.1 MB/s | 8.0 MB 00:01 2024-03-29T11:15:30.666 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - AppStream 1.4 MB/s | 19 MB 00:13 2024-03-29T11:15:34.867 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - CRB 8.5 MB/s | 6.1 MB 00:00 2024-03-29T11:15:36.401 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - Extras packages 84 kB/s | 16 kB 00:00 2024-03-29T11:15:38.755 INFO:teuthology.orchestra.run.smithi094.stdout:Extra Packages for Enterprise Linux 9.3 MB/s | 21 MB 00:02 2024-03-29T11:15:39.884 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - AppStream 875 kB/s | 19 MB 00:22 2024-03-29T11:15:43.777 INFO:teuthology.orchestra.run.smithi094.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-03-29T11:15:46.376 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T11:15:46.394 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T11:15:46.394 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-03-29T11:15:46.394 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T11:15:46.394 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-29T11:15:46.394 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:15:46.394 INFO:teuthology.orchestra.run.smithi094.stdout:Upgrading: 2024-03-29T11:15:46.394 INFO:teuthology.orchestra.run.smithi094.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:15:46.394 INFO:teuthology.orchestra.run.smithi094.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:15:46.395 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-03-29T11:15:46.395 INFO:teuthology.orchestra.run.smithi094.stdout: boost-program-options 2024-03-29T11:15:46.395 INFO:teuthology.orchestra.run.smithi094.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:15:46.395 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:15:46.395 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:15:46.395 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:15:46.395 INFO:teuthology.orchestra.run.smithi094.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:15:46.395 INFO:teuthology.orchestra.run.smithi094.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:15:46.395 INFO:teuthology.orchestra.run.smithi094.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:15:46.395 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:15:46.395 INFO:teuthology.orchestra.run.smithi094.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:15:46.395 INFO:teuthology.orchestra.run.smithi094.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:15:46.395 INFO:teuthology.orchestra.run.smithi094.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:15:46.396 INFO:teuthology.orchestra.run.smithi094.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:15:46.396 INFO:teuthology.orchestra.run.smithi094.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:15:46.396 INFO:teuthology.orchestra.run.smithi094.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:15:46.396 INFO:teuthology.orchestra.run.smithi094.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:15:46.396 INFO:teuthology.orchestra.run.smithi094.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:15:46.396 INFO:teuthology.orchestra.run.smithi094.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:15:46.396 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-argparse 2024-03-29T11:15:46.396 INFO:teuthology.orchestra.run.smithi094.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:15:46.396 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:15:46.396 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:15:46.396 INFO:teuthology.orchestra.run.smithi094.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:15:46.396 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:15:46.397 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:15:46.397 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:15:46.397 INFO:teuthology.orchestra.run.smithi094.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:15:46.397 INFO:teuthology.orchestra.run.smithi094.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:15:46.397 INFO:teuthology.orchestra.run.smithi094.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:15:46.397 INFO:teuthology.orchestra.run.smithi094.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:15:46.397 INFO:teuthology.orchestra.run.smithi094.stdout:Installing weak dependencies: 2024-03-29T11:15:46.397 INFO:teuthology.orchestra.run.smithi094.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:15:46.397 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:15:46.397 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-29T11:15:46.397 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T11:15:46.397 INFO:teuthology.orchestra.run.smithi094.stdout:Install 30 Packages 2024-03-29T11:15:46.397 INFO:teuthology.orchestra.run.smithi094.stdout:Upgrade 2 Packages 2024-03-29T11:15:46.398 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:15:46.398 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 58 M 2024-03-29T11:15:46.398 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-29T11:15:49.362 INFO:teuthology.orchestra.run.smithi094.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 1.9 MB/s | 5.3 MB 00:02 2024-03-29T11:15:49.470 INFO:teuthology.orchestra.run.smithi094.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 232 kB/s | 25 kB 00:00 2024-03-29T11:15:49.754 INFO:teuthology.orchestra.run.smithi094.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 2.6 MB/s | 742 kB 00:00 2024-03-29T11:15:49.996 INFO:teuthology.orchestra.run.smithi094.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 2.0 MB/s | 498 kB 00:00 2024-03-29T11:15:51.446 INFO:teuthology.orchestra.run.smithi094.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 2.0 MB/s | 9.7 MB 00:04 2024-03-29T11:15:51.467 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - CRB 798 kB/s | 6.1 MB 00:07 2024-03-29T11:15:51.654 INFO:teuthology.orchestra.run.smithi094.stdout:(6/32): ceph-common-19.0.0-1439.g5dcb190d.el9.x 3.8 MB/s | 20 MB 00:05 2024-03-29T11:15:51.679 INFO:teuthology.orchestra.run.smithi094.stdout:(7/32): python3-ceph-argparse-19.0.0-1439.g5dcb 193 kB/s | 45 kB 00:00 2024-03-29T11:15:51.790 INFO:teuthology.orchestra.run.smithi094.stdout:(8/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 2.8 MB/s | 5.1 MB 00:01 2024-03-29T11:15:51.813 INFO:teuthology.orchestra.run.smithi094.stdout:(9/32): python3-ceph-common-19.0.0-1439.g5dcb19 828 kB/s | 131 kB 00:00 2024-03-29T11:15:51.864 INFO:teuthology.orchestra.run.smithi094.stdout:(10/32): python3-cephfs-19.0.0-1439.g5dcb190d.e 880 kB/s | 162 kB 00:00 2024-03-29T11:15:51.956 INFO:teuthology.orchestra.run.smithi094.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.1 MB/s | 99 kB 00:00 2024-03-29T11:15:51.981 INFO:teuthology.orchestra.run.smithi094.stdout:(12/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 1.7 MB/s | 298 kB 00:00 2024-03-29T11:15:52.073 INFO:teuthology.orchestra.run.smithi094.stdout:(13/32): python3-rados-19.0.0-1439.g5dcb190d.el 1.1 MB/s | 320 kB 00:00 2024-03-29T11:15:52.115 INFO:teuthology.orchestra.run.smithi094.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 666 kB/s | 106 kB 00:00 2024-03-29T11:15:52.140 INFO:teuthology.orchestra.run.smithi094.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 159 kB 00:00 2024-03-29T11:15:52.165 INFO:teuthology.orchestra.run.smithi094.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 487 kB/s | 44 kB 00:00 2024-03-29T11:15:52.190 INFO:teuthology.orchestra.run.smithi094.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-29T11:15:52.215 INFO:teuthology.orchestra.run.smithi094.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 841 kB/s | 41 kB 00:00 2024-03-29T11:15:52.242 INFO:teuthology.orchestra.run.smithi094.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-29T11:15:52.295 INFO:teuthology.orchestra.run.smithi094.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.0 MB/s | 220 kB 00:00 2024-03-29T11:15:52.316 INFO:teuthology.orchestra.run.smithi094.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 349 kB/s | 35 kB 00:00 2024-03-29T11:15:52.333 INFO:teuthology.orchestra.run.smithi094.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.2 MB/s | 46 kB 00:00 2024-03-29T11:15:52.458 INFO:teuthology.orchestra.run.smithi094.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 208 kB/s | 26 kB 00:00 2024-03-29T11:15:52.559 INFO:teuthology.orchestra.run.smithi094.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 486 kB/s | 49 kB 00:00 2024-03-29T11:15:52.601 INFO:teuthology.orchestra.run.smithi094.stdout:(25/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.0 MB/s | 309 kB 00:00 2024-03-29T11:15:52.651 INFO:teuthology.orchestra.run.smithi094.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 732 kB/s | 67 kB 00:00 2024-03-29T11:15:52.793 INFO:teuthology.orchestra.run.smithi094.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.3 MB/s | 838 kB 00:00 2024-03-29T11:15:52.818 INFO:teuthology.orchestra.run.smithi094.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-03-29T11:15:52.918 INFO:teuthology.orchestra.run.smithi094.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 7.3 MB/s | 4.4 MB 00:00 2024-03-29T11:15:52.969 INFO:teuthology.orchestra.run.smithi094.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 9.0 MB/s | 1.6 MB 00:00 2024-03-29T11:15:53.917 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 16 kB 00:01 2024-03-29T11:15:54.019 INFO:teuthology.orchestra.run.smithi094.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 2.8 MB/s | 3.1 MB 00:01 2024-03-29T11:15:54.102 INFO:teuthology.orchestra.run.smithi094.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 2.6 MB/s | 3.4 MB 00:01 2024-03-29T11:15:54.107 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:15:54.107 INFO:teuthology.orchestra.run.smithi094.stdout:Total 7.5 MB/s | 58 MB 00:07 2024-03-29T11:15:54.107 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-29T11:15:54.146 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-29T11:15:54.146 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-29T11:15:54.620 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-29T11:15:54.620 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-29T11:15:55.513 INFO:teuthology.orchestra.run.smithi017.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-29T11:15:56.084 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-29T11:15:56.162 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:15:56.191 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:15:56.216 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:15:56.238 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:15:56.255 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:15:56.448 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:15:56.502 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:15:56.818 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:15:56.930 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:15:57.024 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:15:57.112 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:15:57.152 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:15:57.214 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:15:57.243 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:15:57.295 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:15:57.362 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:15:57.387 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:15:57.451 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:15:57.486 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:15:57.593 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:15:57.629 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:15:57.682 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:15:57.710 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:15:57.738 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:15:57.772 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:15:58.082 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:15:58.118 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:15:58.215 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:15:58.247 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:15:58.292 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:15:58.745 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:15:58.770 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:15:58.841 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:15:58.908 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:15:58.939 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:16:00.644 INFO:teuthology.orchestra.run.smithi017.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-29T11:16:01.116 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:16:01.152 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:16:01.697 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:16:01.730 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:16:01.821 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:16:01.821 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:16:01.821 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:16:01.821 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:16:01.856 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:16:03.356 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-03-29T11:16:03.364 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-03-29T11:16:03.364 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repository Size 2024-03-29T11:16:03.364 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-03-29T11:16:03.364 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-03-29T11:16:03.364 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:16:03.364 INFO:teuthology.orchestra.run.smithi017.stdout:Upgrading: 2024-03-29T11:16:03.364 INFO:teuthology.orchestra.run.smithi017.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:16:03.365 INFO:teuthology.orchestra.run.smithi017.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:16:03.365 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2024-03-29T11:16:03.365 INFO:teuthology.orchestra.run.smithi017.stdout: boost-program-options 2024-03-29T11:16:03.365 INFO:teuthology.orchestra.run.smithi017.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:16:03.365 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:16:03.365 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:16:03.365 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:16:03.365 INFO:teuthology.orchestra.run.smithi017.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:16:03.365 INFO:teuthology.orchestra.run.smithi017.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:16:03.365 INFO:teuthology.orchestra.run.smithi017.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:16:03.365 INFO:teuthology.orchestra.run.smithi017.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:16:03.365 INFO:teuthology.orchestra.run.smithi017.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:16:03.365 INFO:teuthology.orchestra.run.smithi017.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:16:03.366 INFO:teuthology.orchestra.run.smithi017.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:16:03.366 INFO:teuthology.orchestra.run.smithi017.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:16:03.366 INFO:teuthology.orchestra.run.smithi017.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:16:03.366 INFO:teuthology.orchestra.run.smithi017.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:16:03.366 INFO:teuthology.orchestra.run.smithi017.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:16:03.366 INFO:teuthology.orchestra.run.smithi017.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:16:03.366 INFO:teuthology.orchestra.run.smithi017.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:16:03.366 INFO:teuthology.orchestra.run.smithi017.stdout: python3-ceph-argparse 2024-03-29T11:16:03.366 INFO:teuthology.orchestra.run.smithi017.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:16:03.366 INFO:teuthology.orchestra.run.smithi017.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:16:03.366 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:16:03.366 INFO:teuthology.orchestra.run.smithi017.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:16:03.366 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:16:03.366 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:16:03.367 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:16:03.367 INFO:teuthology.orchestra.run.smithi017.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:16:03.367 INFO:teuthology.orchestra.run.smithi017.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:16:03.367 INFO:teuthology.orchestra.run.smithi017.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:16:03.367 INFO:teuthology.orchestra.run.smithi017.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:16:03.367 INFO:teuthology.orchestra.run.smithi017.stdout:Installing weak dependencies: 2024-03-29T11:16:03.367 INFO:teuthology.orchestra.run.smithi017.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:16:03.367 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:16:03.367 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-03-29T11:16:03.367 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-03-29T11:16:03.367 INFO:teuthology.orchestra.run.smithi017.stdout:Install 30 Packages 2024-03-29T11:16:03.367 INFO:teuthology.orchestra.run.smithi017.stdout:Upgrade 2 Packages 2024-03-29T11:16:03.367 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:16:03.368 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 58 M 2024-03-29T11:16:03.368 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-03-29T11:16:04.362 INFO:teuthology.orchestra.run.smithi017.stdout:(1/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 13 MB/s | 9.7 MB 00:00 2024-03-29T11:16:04.412 INFO:teuthology.orchestra.run.smithi017.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 506 kB/s | 25 kB 00:00 2024-03-29T11:16:04.479 INFO:teuthology.orchestra.run.smithi017.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 11 MB/s | 742 kB 00:00 2024-03-29T11:16:04.529 INFO:teuthology.orchestra.run.smithi017.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 9.7 MB/s | 498 kB 00:00 2024-03-29T11:16:04.813 INFO:teuthology.orchestra.run.smithi017.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 18 MB/s | 5.1 MB 00:00 2024-03-29T11:16:04.855 INFO:teuthology.orchestra.run.smithi017.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 1.1 MB/s | 45 kB 00:00 2024-03-29T11:16:04.913 INFO:teuthology.orchestra.run.smithi017.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 2.2 MB/s | 131 kB 00:00 2024-03-29T11:16:04.972 INFO:teuthology.orchestra.run.smithi017.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 2.7 MB/s | 162 kB 00:00 2024-03-29T11:16:05.031 INFO:teuthology.orchestra.run.smithi017.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 5.4 MB/s | 320 kB 00:00 2024-03-29T11:16:05.089 INFO:teuthology.orchestra.run.smithi017.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 5.0 MB/s | 298 kB 00:00 2024-03-29T11:16:05.131 INFO:teuthology.orchestra.run.smithi017.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 2.3 MB/s | 99 kB 00:00 2024-03-29T11:16:05.231 INFO:teuthology.orchestra.run.smithi017.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 1.0 MB/s | 106 kB 00:00 2024-03-29T11:16:05.273 INFO:teuthology.orchestra.run.smithi017.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-29T11:16:05.307 INFO:teuthology.orchestra.run.smithi017.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-29T11:16:05.366 INFO:teuthology.orchestra.run.smithi017.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-29T11:16:05.399 INFO:teuthology.orchestra.run.smithi017.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 8.3 MB/s | 282 kB 00:00 2024-03-29T11:16:05.433 INFO:teuthology.orchestra.run.smithi017.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-29T11:16:05.466 INFO:teuthology.orchestra.run.smithi017.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 6.5 MB/s | 220 kB 00:00 2024-03-29T11:16:05.500 INFO:teuthology.orchestra.run.smithi017.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 1.0 MB/s | 35 kB 00:00 2024-03-29T11:16:05.584 INFO:teuthology.orchestra.run.smithi017.stdout:(20/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x8 2.7 MB/s | 5.3 MB 00:01 2024-03-29T11:16:05.609 INFO:teuthology.orchestra.run.smithi017.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 426 kB/s | 46 kB 00:00 2024-03-29T11:16:05.918 INFO:teuthology.orchestra.run.smithi017.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 926 kB/s | 309 kB 00:00 2024-03-29T11:16:05.960 INFO:teuthology.orchestra.run.smithi017.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 634 kB/s | 26 kB 00:00 2024-03-29T11:16:06.002 INFO:teuthology.orchestra.run.smithi017.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-03-29T11:16:06.069 INFO:teuthology.orchestra.run.smithi017.stdout:(25/32): libarrow-9.0.0-10.el9.x86_64.rpm 9.6 MB/s | 4.4 MB 00:00 2024-03-29T11:16:06.094 INFO:teuthology.orchestra.run.smithi017.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 757 kB/s | 67 kB 00:00 2024-03-29T11:16:06.278 INFO:teuthology.orchestra.run.smithi017.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.9 MB/s | 838 kB 00:00 2024-03-29T11:16:06.328 INFO:teuthology.orchestra.run.smithi017.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 822 kB/s | 192 kB 00:00 2024-03-29T11:16:06.561 INFO:teuthology.orchestra.run.smithi017.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 5.6 MB/s | 1.6 MB 00:00 2024-03-29T11:16:07.153 INFO:teuthology.orchestra.run.smithi017.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 5.3 MB/s | 3.1 MB 00:00 2024-03-29T11:16:07.287 INFO:teuthology.orchestra.run.smithi017.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 3.5 MB/s | 3.4 MB 00:00 2024-03-29T11:16:07.912 INFO:teuthology.orchestra.run.smithi017.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 4.6 MB/s | 20 MB 00:04 2024-03-29T11:16:07.917 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:16:07.918 INFO:teuthology.orchestra.run.smithi017.stdout:Total 13 MB/s | 58 MB 00:04 2024-03-29T11:16:07.918 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-03-29T11:16:07.960 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-03-29T11:16:07.960 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-03-29T11:16:08.438 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-03-29T11:16:08.438 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-03-29T11:16:10.120 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-03-29T11:16:10.198 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:16:10.225 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:16:10.250 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:16:10.273 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:16:10.289 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:16:10.457 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:16:10.512 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:16:10.864 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:16:10.935 INFO:teuthology.orchestra.run.smithi017.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:16:11.027 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:16:11.133 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:16:11.161 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:16:11.232 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:16:11.261 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:16:11.305 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:16:11.375 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:16:11.397 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:16:11.421 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:16:11.421 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /sys 2024-03-29T11:16:11.421 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /proc 2024-03-29T11:16:11.421 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /mnt 2024-03-29T11:16:11.422 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /var/tmp 2024-03-29T11:16:11.422 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /home 2024-03-29T11:16:11.422 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /root 2024-03-29T11:16:11.422 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /tmp 2024-03-29T11:16:11.422 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:16:11.444 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:16:11.478 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:16:11.591 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:16:11.630 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:16:11.671 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:16:11.703 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:16:11.732 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:16:11.765 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:16:12.090 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:16:12.119 INFO:teuthology.orchestra.run.smithi017.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:16:12.210 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:16:12.240 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:16:12.286 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:16:12.330 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:16:12.354 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:16:12.354 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:16:12.354 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:16:12.355 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:16:12.355 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:16:12.355 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:16:12.357 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:16:12.441 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:16:12.441 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:16:12.765 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:16:12.797 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:16:12.868 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:16:12.933 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:16:12.966 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:16:13.610 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:16:13.610 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:16:13.610 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:16:13.610 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:16:13.610 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:16:13.610 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:16:13.611 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:16:13.611 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:16:13.611 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:16:13.611 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:16:13.611 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:16:13.611 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:16:13.611 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:16:13.611 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:16:13.611 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:16:13.611 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:16:13.611 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:16:13.611 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:16:13.611 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:16:13.612 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:16:13.612 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:16:13.612 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:16:13.612 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:16:13.612 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:16:13.612 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:16:13.612 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:16:13.612 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:16:13.612 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:16:13.612 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:16:13.612 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:16:13.612 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:16:13.612 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:16:13.612 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:16:13.613 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:16:14.577 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:16:14.577 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:16:14.577 INFO:teuthology.orchestra.run.smithi094.stdout:Upgraded: 2024-03-29T11:16:14.578 INFO:teuthology.orchestra.run.smithi094.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:14.578 INFO:teuthology.orchestra.run.smithi094.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:14.578 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-29T11:16:14.578 INFO:teuthology.orchestra.run.smithi094.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:16:14.578 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:14.578 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:14.578 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:14.578 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:14.578 INFO:teuthology.orchestra.run.smithi094.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:16:14.578 INFO:teuthology.orchestra.run.smithi094.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:16:14.578 INFO:teuthology.orchestra.run.smithi094.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:16:14.578 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:14.578 INFO:teuthology.orchestra.run.smithi094.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:16:14.578 INFO:teuthology.orchestra.run.smithi094.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:16:14.579 INFO:teuthology.orchestra.run.smithi094.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:16:14.579 INFO:teuthology.orchestra.run.smithi094.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:14.579 INFO:teuthology.orchestra.run.smithi094.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:16:14.579 INFO:teuthology.orchestra.run.smithi094.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:14.579 INFO:teuthology.orchestra.run.smithi094.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:16:14.579 INFO:teuthology.orchestra.run.smithi094.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:16:14.579 INFO:teuthology.orchestra.run.smithi094.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:16:14.579 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:14.579 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:14.579 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:14.579 INFO:teuthology.orchestra.run.smithi094.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:16:14.579 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:14.579 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:14.579 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:14.579 INFO:teuthology.orchestra.run.smithi094.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:16:14.580 INFO:teuthology.orchestra.run.smithi094.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:16:14.580 INFO:teuthology.orchestra.run.smithi094.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:16:14.580 INFO:teuthology.orchestra.run.smithi094.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:16:14.580 INFO:teuthology.orchestra.run.smithi094.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:16:14.580 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:16:14.580 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T11:16:14.947 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-test 2024-03-29T11:16:15.195 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:16:15.237 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:16:15.498 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:32 ago on Fri 29 Mar 2024 11:15:43 AM UTC. 2024-03-29T11:16:15.657 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T11:16:15.658 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T11:16:15.658 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-03-29T11:16:15.658 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T11:16:15.658 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-29T11:16:15.658 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:16:15.658 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-03-29T11:16:15.658 INFO:teuthology.orchestra.run.smithi094.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:16:15.658 INFO:teuthology.orchestra.run.smithi094.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:16:15.658 INFO:teuthology.orchestra.run.smithi094.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:16:15.658 INFO:teuthology.orchestra.run.smithi094.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:16:15.658 INFO:teuthology.orchestra.run.smithi094.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:16:15.659 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:16:15.659 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-29T11:16:15.659 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T11:16:15.659 INFO:teuthology.orchestra.run.smithi094.stdout:Install 6 Packages 2024-03-29T11:16:15.659 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:16:15.659 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 48 M 2024-03-29T11:16:15.659 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 204 M 2024-03-29T11:16:15.660 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-29T11:16:15.821 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:16:15.856 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:16:15.932 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:16:15.932 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:16:15.932 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:16:15.932 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:16:15.974 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:16:16.266 INFO:teuthology.orchestra.run.smithi094.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 639 kB/s | 217 kB 00:00 2024-03-29T11:16:16.391 INFO:teuthology.orchestra.run.smithi094.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 2.3 MB/s | 300 kB 00:00 2024-03-29T11:16:16.458 INFO:teuthology.orchestra.run.smithi094.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 938 kB/s | 63 kB 00:00 2024-03-29T11:16:16.517 INFO:teuthology.orchestra.run.smithi094.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 276 kB/s | 163 kB 00:00 2024-03-29T11:16:16.875 INFO:teuthology.orchestra.run.smithi094.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 449 kB/s | 187 kB 00:00 2024-03-29T11:16:26.037 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:16:26.037 INFO:teuthology.orchestra.run.smithi017.stdout:skipping the directory /sys 2024-03-29T11:16:26.037 INFO:teuthology.orchestra.run.smithi017.stdout:skipping the directory /proc 2024-03-29T11:16:26.037 INFO:teuthology.orchestra.run.smithi017.stdout:skipping the directory /mnt 2024-03-29T11:16:26.038 INFO:teuthology.orchestra.run.smithi017.stdout:skipping the directory /var/tmp 2024-03-29T11:16:26.038 INFO:teuthology.orchestra.run.smithi017.stdout:skipping the directory /home 2024-03-29T11:16:26.038 INFO:teuthology.orchestra.run.smithi017.stdout:skipping the directory /root 2024-03-29T11:16:26.038 INFO:teuthology.orchestra.run.smithi017.stdout:skipping the directory /tmp 2024-03-29T11:16:26.038 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:16:26.973 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:16:26.996 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:16:26.996 INFO:teuthology.orchestra.run.smithi017.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:16:26.997 INFO:teuthology.orchestra.run.smithi017.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:16:26.997 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:16:26.997 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:16:26.997 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:16:26.998 INFO:teuthology.orchestra.run.smithi017.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:16:27.067 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:16:27.067 INFO:teuthology.orchestra.run.smithi017.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:16:28.119 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:16:28.119 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:16:28.119 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:16:28.120 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:16:28.120 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:16:28.120 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:16:28.120 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:16:28.120 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:16:28.120 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:16:28.120 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:16:28.120 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:16:28.120 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:16:28.120 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:16:28.120 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:16:28.120 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:16:28.120 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:16:28.120 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:16:28.121 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:16:28.121 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:16:28.121 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:16:28.121 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:16:28.121 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:16:28.121 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:16:28.121 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:16:28.121 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:16:28.121 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:16:28.121 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:16:28.121 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:16:28.121 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:16:28.121 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:16:28.122 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:16:28.122 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:16:28.122 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:16:28.122 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:16:29.310 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:16:29.310 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:16:29.310 INFO:teuthology.orchestra.run.smithi017.stdout:Upgraded: 2024-03-29T11:16:29.310 INFO:teuthology.orchestra.run.smithi017.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:29.310 INFO:teuthology.orchestra.run.smithi017.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:29.310 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-03-29T11:16:29.310 INFO:teuthology.orchestra.run.smithi017.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:16:29.310 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:29.310 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:29.310 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:29.310 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:29.311 INFO:teuthology.orchestra.run.smithi017.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:16:29.311 INFO:teuthology.orchestra.run.smithi017.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:16:29.311 INFO:teuthology.orchestra.run.smithi017.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:16:29.311 INFO:teuthology.orchestra.run.smithi017.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:29.311 INFO:teuthology.orchestra.run.smithi017.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:16:29.311 INFO:teuthology.orchestra.run.smithi017.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:16:29.311 INFO:teuthology.orchestra.run.smithi017.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:16:29.311 INFO:teuthology.orchestra.run.smithi017.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:29.311 INFO:teuthology.orchestra.run.smithi017.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:16:29.311 INFO:teuthology.orchestra.run.smithi017.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:29.311 INFO:teuthology.orchestra.run.smithi017.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:16:29.311 INFO:teuthology.orchestra.run.smithi017.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:16:29.311 INFO:teuthology.orchestra.run.smithi017.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:16:29.311 INFO:teuthology.orchestra.run.smithi017.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:29.311 INFO:teuthology.orchestra.run.smithi017.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:29.311 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:29.312 INFO:teuthology.orchestra.run.smithi017.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:16:29.312 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:29.312 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:29.312 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:29.312 INFO:teuthology.orchestra.run.smithi017.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:16:29.312 INFO:teuthology.orchestra.run.smithi017.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:16:29.312 INFO:teuthology.orchestra.run.smithi017.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:16:29.312 INFO:teuthology.orchestra.run.smithi017.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:16:29.312 INFO:teuthology.orchestra.run.smithi017.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:16:29.312 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:16:29.312 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-03-29T11:16:29.651 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-test 2024-03-29T11:16:29.850 INFO:teuthology.orchestra.run.smithi094.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 3.4 MB/s | 47 MB 00:13 2024-03-29T11:16:29.852 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:16:29.852 INFO:teuthology.orchestra.run.smithi094.stdout:Total 3.4 MB/s | 48 MB 00:14 2024-03-29T11:16:29.865 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-29T11:16:29.884 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-29T11:16:29.884 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-29T11:16:30.237 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:00:30 ago on Fri 29 Mar 2024 11:16:00 AM UTC. 2024-03-29T11:16:30.266 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-29T11:16:30.266 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-29T11:16:30.399 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-03-29T11:16:30.399 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-03-29T11:16:30.399 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repository Size 2024-03-29T11:16:30.399 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-03-29T11:16:30.400 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-03-29T11:16:30.400 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:16:30.400 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2024-03-29T11:16:30.400 INFO:teuthology.orchestra.run.smithi017.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:16:30.400 INFO:teuthology.orchestra.run.smithi017.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:16:30.400 INFO:teuthology.orchestra.run.smithi017.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:16:30.400 INFO:teuthology.orchestra.run.smithi017.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:16:30.400 INFO:teuthology.orchestra.run.smithi017.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:16:30.400 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:16:30.400 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-03-29T11:16:30.400 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-03-29T11:16:30.400 INFO:teuthology.orchestra.run.smithi017.stdout:Install 6 Packages 2024-03-29T11:16:30.400 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:16:30.401 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 48 M 2024-03-29T11:16:30.401 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 204 M 2024-03-29T11:16:30.401 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-03-29T11:16:30.847 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-29T11:16:30.886 INFO:teuthology.orchestra.run.smithi017.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.0 MB/s | 217 kB 00:00 2024-03-29T11:16:30.925 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:16:30.944 INFO:teuthology.orchestra.run.smithi017.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-29T11:16:30.967 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:16:30.978 INFO:teuthology.orchestra.run.smithi017.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-29T11:16:31.008 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:16:31.028 INFO:teuthology.orchestra.run.smithi017.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 457 kB/s | 163 kB 00:00 2024-03-29T11:16:31.056 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:16:31.070 INFO:teuthology.orchestra.run.smithi017.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 2.0 MB/s | 187 kB 00:00 2024-03-29T11:16:31.086 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:16:35.562 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:16:35.637 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:16:36.845 INFO:teuthology.orchestra.run.smithi017.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 7.7 MB/s | 47 MB 00:06 2024-03-29T11:16:36.847 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:16:36.848 INFO:teuthology.orchestra.run.smithi017.stdout:Total 7.5 MB/s | 48 MB 00:06 2024-03-29T11:16:36.861 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-03-29T11:16:36.881 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-03-29T11:16:36.881 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-03-29T11:16:37.279 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-03-29T11:16:37.279 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-03-29T11:16:37.870 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-03-29T11:16:37.949 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:16:37.951 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:16:37.951 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:16:37.952 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:16:37.952 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:16:37.952 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:16:37.952 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:16:37.987 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:16:38.027 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:16:38.079 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:16:38.109 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:16:38.329 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:16:38.329 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:16:38.329 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-29T11:16:38.329 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:38.329 INFO:teuthology.orchestra.run.smithi094.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:16:38.329 INFO:teuthology.orchestra.run.smithi094.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:38.329 INFO:teuthology.orchestra.run.smithi094.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:16:38.329 INFO:teuthology.orchestra.run.smithi094.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:16:38.329 INFO:teuthology.orchestra.run.smithi094.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:16:38.329 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:16:38.330 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T11:16:38.664 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph 2024-03-29T11:16:39.231 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:56 ago on Fri 29 Mar 2024 11:15:43 AM UTC. 2024-03-29T11:16:39.384 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T11:16:39.386 INFO:teuthology.orchestra.run.smithi094.stdout:========================================================================================= 2024-03-29T11:16:39.386 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-03-29T11:16:39.386 INFO:teuthology.orchestra.run.smithi094.stdout:========================================================================================= 2024-03-29T11:16:39.386 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-29T11:16:39.386 INFO:teuthology.orchestra.run.smithi094.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:16:39.387 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-03-29T11:16:39.387 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:16:39.387 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:16:39.387 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:16:39.387 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:16:39.387 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:16:39.387 INFO:teuthology.orchestra.run.smithi094.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:16:39.387 INFO:teuthology.orchestra.run.smithi094.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:16:39.387 INFO:teuthology.orchestra.run.smithi094.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:16:39.387 INFO:teuthology.orchestra.run.smithi094.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:16:39.387 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:16:39.387 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:16:39.387 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:16:39.387 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:16:39.388 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:16:39.388 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:16:39.388 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:16:39.388 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:16:39.388 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:16:39.388 INFO:teuthology.orchestra.run.smithi094.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:16:39.388 INFO:teuthology.orchestra.run.smithi094.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:16:39.388 INFO:teuthology.orchestra.run.smithi094.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:16:39.388 INFO:teuthology.orchestra.run.smithi094.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:16:39.388 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:16:39.388 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:16:39.388 INFO:teuthology.orchestra.run.smithi094.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:16:39.388 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:16:39.388 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:16:39.389 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:16:39.389 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:16:39.389 INFO:teuthology.orchestra.run.smithi094.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:16:39.389 INFO:teuthology.orchestra.run.smithi094.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:16:39.389 INFO:teuthology.orchestra.run.smithi094.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:16:39.389 INFO:teuthology.orchestra.run.smithi094.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:16:39.389 INFO:teuthology.orchestra.run.smithi094.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:16:39.389 INFO:teuthology.orchestra.run.smithi094.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:16:39.389 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:16:39.389 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-29T11:16:39.389 INFO:teuthology.orchestra.run.smithi094.stdout:========================================================================================= 2024-03-29T11:16:39.389 INFO:teuthology.orchestra.run.smithi094.stdout:Install 36 Packages 2024-03-29T11:16:39.389 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:16:39.390 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 30 M 2024-03-29T11:16:39.390 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 107 M 2024-03-29T11:16:39.390 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-29T11:16:40.471 INFO:teuthology.orchestra.run.smithi094.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 16 kB/s | 6.6 kB 00:00 2024-03-29T11:16:41.472 INFO:teuthology.orchestra.run.smithi094.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 1.1 MB/s | 1.5 MB 00:01 2024-03-29T11:16:41.955 INFO:teuthology.orchestra.run.smithi094.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 1.1 MB/s | 2.1 MB 00:01 2024-03-29T11:16:42.646 INFO:teuthology.orchestra.run.smithi094.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 358 kB/s | 247 kB 00:00 2024-03-29T11:16:42.764 INFO:teuthology.orchestra.run.smithi094.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-29T11:16:42.805 INFO:teuthology.orchestra.run.smithi094.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-29T11:16:42.815 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:16:42.839 INFO:teuthology.orchestra.run.smithi094.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 948 kB/s | 32 kB 00:00 2024-03-29T11:16:42.889 INFO:teuthology.orchestra.run.smithi094.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 945 kB/s | 47 kB 00:00 2024-03-29T11:16:42.894 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:16:42.923 INFO:teuthology.orchestra.run.smithi094.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-29T11:16:42.995 INFO:teuthology.orchestra.run.smithi094.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.3 MB/s | 241 kB 00:00 2024-03-29T11:16:43.065 INFO:teuthology.orchestra.run.smithi094.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 16 MB/s | 1.1 MB 00:00 2024-03-29T11:16:43.099 INFO:teuthology.orchestra.run.smithi094.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-29T11:16:43.133 INFO:teuthology.orchestra.run.smithi094.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-29T11:16:43.166 INFO:teuthology.orchestra.run.smithi094.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-29T11:16:43.308 INFO:teuthology.orchestra.run.smithi094.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 884 kB/s | 125 kB 00:00 2024-03-29T11:16:43.467 INFO:teuthology.orchestra.run.smithi094.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 141 kB/s | 22 kB 00:00 2024-03-29T11:16:43.826 INFO:teuthology.orchestra.run.smithi094.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 421 kB/s | 151 kB 00:00 2024-03-29T11:16:43.909 INFO:teuthology.orchestra.run.smithi094.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 521 kB/s | 43 kB 00:00 2024-03-29T11:16:44.018 INFO:teuthology.orchestra.run.smithi094.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.5 MB/s | 172 kB 00:00 2024-03-29T11:16:44.152 INFO:teuthology.orchestra.run.smithi094.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.6 MB/s | 358 kB 00:00 2024-03-29T11:16:44.219 INFO:teuthology.orchestra.run.smithi094.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 160 kB/s | 11 kB 00:00 2024-03-29T11:16:44.310 INFO:teuthology.orchestra.run.smithi094.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 193 kB/s | 18 kB 00:00 2024-03-29T11:16:44.378 INFO:teuthology.orchestra.run.smithi094.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 347 kB/s | 23 kB 00:00 2024-03-29T11:16:44.461 INFO:teuthology.orchestra.run.smithi094.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 233 kB/s | 19 kB 00:00 2024-03-29T11:16:44.536 INFO:teuthology.orchestra.run.smithi094.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 262 kB/s | 20 kB 00:00 2024-03-29T11:16:44.628 INFO:teuthology.orchestra.run.smithi094.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 505 kB/s | 46 kB 00:00 2024-03-29T11:16:44.712 INFO:teuthology.orchestra.run.smithi094.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 948 kB/s | 79 kB 00:00 2024-03-29T11:16:44.795 INFO:teuthology.orchestra.run.smithi094.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-03-29T11:16:44.862 INFO:teuthology.orchestra.run.smithi094.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 247 kB/s | 16 kB 00:00 2024-03-29T11:16:44.938 INFO:teuthology.orchestra.run.smithi094.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-03-29T11:16:45.005 INFO:teuthology.orchestra.run.smithi094.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 535 kB/s | 36 kB 00:00 2024-03-29T11:16:45.089 INFO:teuthology.orchestra.run.smithi094.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-03-29T11:16:45.197 INFO:teuthology.orchestra.run.smithi094.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.9 MB/s | 427 kB 00:00 2024-03-29T11:16:45.272 INFO:teuthology.orchestra.run.smithi094.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 266 kB/s | 20 kB 00:00 2024-03-29T11:16:45.286 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:16:45.286 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:16:45.286 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:16:45.286 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:16:45.286 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:16:45.286 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:16:45.640 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:16:45.640 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:16:45.640 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-03-29T11:16:45.640 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:45.640 INFO:teuthology.orchestra.run.smithi017.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:16:45.640 INFO:teuthology.orchestra.run.smithi017.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:45.640 INFO:teuthology.orchestra.run.smithi017.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:16:45.640 INFO:teuthology.orchestra.run.smithi017.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:16:45.641 INFO:teuthology.orchestra.run.smithi017.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:16:45.641 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:16:45.641 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-03-29T11:16:45.983 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph 2024-03-29T11:16:46.553 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:00:46 ago on Fri 29 Mar 2024 11:16:00 AM UTC. 2024-03-29T11:16:46.713 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-03-29T11:16:46.715 INFO:teuthology.orchestra.run.smithi017.stdout:========================================================================================= 2024-03-29T11:16:46.715 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repository Size 2024-03-29T11:16:46.715 INFO:teuthology.orchestra.run.smithi017.stdout:========================================================================================= 2024-03-29T11:16:46.715 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-03-29T11:16:46.715 INFO:teuthology.orchestra.run.smithi017.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:16:46.716 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2024-03-29T11:16:46.716 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:16:46.716 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:16:46.716 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:16:46.716 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:16:46.716 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:16:46.716 INFO:teuthology.orchestra.run.smithi017.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:16:46.716 INFO:teuthology.orchestra.run.smithi017.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:16:46.716 INFO:teuthology.orchestra.run.smithi017.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:16:46.716 INFO:teuthology.orchestra.run.smithi017.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:16:46.716 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:16:46.716 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:16:46.716 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:16:46.716 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:16:46.717 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:16:46.717 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:16:46.717 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:16:46.717 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:16:46.717 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:16:46.717 INFO:teuthology.orchestra.run.smithi017.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:16:46.717 INFO:teuthology.orchestra.run.smithi017.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:16:46.717 INFO:teuthology.orchestra.run.smithi017.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:16:46.717 INFO:teuthology.orchestra.run.smithi017.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:16:46.717 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:16:46.717 INFO:teuthology.orchestra.run.smithi017.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:16:46.717 INFO:teuthology.orchestra.run.smithi017.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:16:46.717 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:16:46.717 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:16:46.718 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:16:46.718 INFO:teuthology.orchestra.run.smithi017.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:16:46.718 INFO:teuthology.orchestra.run.smithi017.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:16:46.718 INFO:teuthology.orchestra.run.smithi017.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:16:46.718 INFO:teuthology.orchestra.run.smithi017.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:16:46.718 INFO:teuthology.orchestra.run.smithi017.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:16:46.718 INFO:teuthology.orchestra.run.smithi017.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:16:46.718 INFO:teuthology.orchestra.run.smithi017.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:16:46.718 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:16:46.718 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-03-29T11:16:46.718 INFO:teuthology.orchestra.run.smithi017.stdout:========================================================================================= 2024-03-29T11:16:46.718 INFO:teuthology.orchestra.run.smithi017.stdout:Install 36 Packages 2024-03-29T11:16:46.718 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:16:46.719 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 30 M 2024-03-29T11:16:46.719 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 107 M 2024-03-29T11:16:46.719 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-03-29T11:16:47.343 INFO:teuthology.orchestra.run.smithi017.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 18 kB/s | 6.6 kB 00:00 2024-03-29T11:16:49.093 INFO:teuthology.orchestra.run.smithi017.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 718 kB/s | 1.5 MB 00:02 2024-03-29T11:16:50.685 INFO:teuthology.orchestra.run.smithi017.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 585 kB/s | 2.1 MB 00:03 2024-03-29T11:16:51.068 INFO:teuthology.orchestra.run.smithi017.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 646 kB/s | 247 kB 00:00 2024-03-29T11:16:51.169 INFO:teuthology.orchestra.run.smithi017.stdout:(5/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 1.2 MB/s | 4.7 MB 00:03 2024-03-29T11:16:51.227 INFO:teuthology.orchestra.run.smithi017.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.5 MB/s | 144 kB 00:00 2024-03-29T11:16:51.252 INFO:teuthology.orchestra.run.smithi017.stdout:(7/36): lua-5.4.4-4.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-03-29T11:16:51.278 INFO:teuthology.orchestra.run.smithi017.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 633 kB/s | 32 kB 00:00 2024-03-29T11:16:51.303 INFO:teuthology.orchestra.run.smithi017.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 949 kB/s | 47 kB 00:00 2024-03-29T11:16:51.328 INFO:teuthology.orchestra.run.smithi017.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 773 kB/s | 39 kB 00:00 2024-03-29T11:16:51.353 INFO:teuthology.orchestra.run.smithi017.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-29T11:16:51.404 INFO:teuthology.orchestra.run.smithi017.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-29T11:16:51.429 INFO:teuthology.orchestra.run.smithi017.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 103 kB 00:00 2024-03-29T11:16:51.454 INFO:teuthology.orchestra.run.smithi017.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-29T11:16:51.480 INFO:teuthology.orchestra.run.smithi017.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-29T11:16:51.580 INFO:teuthology.orchestra.run.smithi017.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 223 kB/s | 22 kB 00:00 2024-03-29T11:16:51.689 INFO:teuthology.orchestra.run.smithi017.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 534 kB/s | 125 kB 00:00 2024-03-29T11:16:51.964 INFO:teuthology.orchestra.run.smithi017.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 394 kB/s | 151 kB 00:00 2024-03-29T11:16:51.989 INFO:teuthology.orchestra.run.smithi017.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 145 kB/s | 43 kB 00:00 2024-03-29T11:16:52.098 INFO:teuthology.orchestra.run.smithi017.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-03-29T11:16:52.165 INFO:teuthology.orchestra.run.smithi017.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 161 kB/s | 11 kB 00:00 2024-03-29T11:16:52.232 INFO:teuthology.orchestra.run.smithi017.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 266 kB/s | 18 kB 00:00 2024-03-29T11:16:52.257 INFO:teuthology.orchestra.run.smithi017.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-03-29T11:16:52.299 INFO:teuthology.orchestra.run.smithi017.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 346 kB/s | 23 kB 00:00 2024-03-29T11:16:52.324 INFO:teuthology.orchestra.run.smithi017.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 291 kB/s | 19 kB 00:00 2024-03-29T11:16:52.366 INFO:teuthology.orchestra.run.smithi017.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 294 kB/s | 20 kB 00:00 2024-03-29T11:16:52.391 INFO:teuthology.orchestra.run.smithi017.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 693 kB/s | 46 kB 00:00 2024-03-29T11:16:52.450 INFO:teuthology.orchestra.run.smithi017.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 944 kB/s | 79 kB 00:00 2024-03-29T11:16:52.492 INFO:teuthology.orchestra.run.smithi017.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.7 MB/s | 272 kB 00:00 2024-03-29T11:16:52.517 INFO:teuthology.orchestra.run.smithi017.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 245 kB/s | 16 kB 00:00 2024-03-29T11:16:52.576 INFO:teuthology.orchestra.run.smithi017.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.1 MB/s | 90 kB 00:00 2024-03-29T11:16:52.601 INFO:teuthology.orchestra.run.smithi017.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 427 kB/s | 36 kB 00:00 2024-03-29T11:16:52.668 INFO:teuthology.orchestra.run.smithi017.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-03-29T11:16:52.727 INFO:teuthology.orchestra.run.smithi017.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.3 MB/s | 427 kB 00:00 2024-03-29T11:16:52.752 INFO:teuthology.orchestra.run.smithi017.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 239 kB/s | 20 kB 00:00 2024-03-29T11:16:54.347 INFO:teuthology.orchestra.run.smithi094.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 346 kB/s | 4.7 MB 00:13 2024-03-29T11:17:00.980 INFO:teuthology.orchestra.run.smithi094.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 881 kB/s | 17 MB 00:19 2024-03-29T11:17:00.985 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:00.985 INFO:teuthology.orchestra.run.smithi094.stdout:Total 1.4 MB/s | 30 MB 00:21 2024-03-29T11:17:01.009 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-29T11:17:01.051 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-29T11:17:01.051 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-29T11:17:01.456 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-29T11:17:01.457 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-29T11:17:02.812 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-29T11:17:02.874 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:17:02.913 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:17:02.943 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:17:03.014 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:17:03.050 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:17:03.083 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:17:03.175 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:17:03.253 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:17:03.294 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:17:03.366 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:17:03.420 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:17:03.467 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:17:03.516 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:17:03.571 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:17:03.756 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:17:03.804 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:17:03.843 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:17:03.878 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:17:03.908 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:17:03.984 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:17:04.018 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:17:04.052 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:17:04.092 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:17:04.127 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:17:04.170 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:17:04.259 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:17:04.301 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:17:04.373 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:17:04.509 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:17:04.627 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:17:04.795 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:17:04.823 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:17:04.846 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:17:04.846 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:04.846 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:17:04.846 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:17:04.847 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:17:04.847 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:17:05.409 INFO:teuthology.orchestra.run.smithi017.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 1.0 MB/s | 17 MB 00:16 2024-03-29T11:17:05.415 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:05.415 INFO:teuthology.orchestra.run.smithi017.stdout:Total 1.6 MB/s | 30 MB 00:18 2024-03-29T11:17:05.439 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-03-29T11:17:05.485 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-03-29T11:17:05.486 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-03-29T11:17:05.915 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-03-29T11:17:05.915 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-03-29T11:17:06.358 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:17:06.385 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:17:06.386 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:06.386 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:17:06.386 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:17:06.386 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:17:06.386 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:17:06.842 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:17:06.865 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:17:06.865 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:06.865 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:17:06.865 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:17:06.866 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:17:06.866 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:17:07.084 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:17:07.107 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:17:07.107 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:07.107 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:17:07.107 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:17:07.107 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:17:07.107 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:17:07.174 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:17:07.414 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-03-29T11:17:07.479 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:17:07.528 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:17:07.556 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:17:07.631 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:17:07.672 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:17:07.705 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:17:07.800 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:17:07.885 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:17:07.933 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:17:08.008 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:17:08.067 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:17:08.105 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:17:08.154 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:17:08.210 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:17:08.408 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:17:08.460 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:17:08.498 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:17:08.533 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:17:08.562 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:17:08.641 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:17:08.673 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:17:08.706 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:17:08.746 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:17:08.782 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:17:08.816 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:17:08.844 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:17:08.844 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:17:08.844 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:17:08.844 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:17:08.844 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:17:08.844 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:17:08.844 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:17:08.844 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:17:08.844 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:17:08.844 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:17:08.844 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:17:08.844 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:17:08.844 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:17:08.844 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:17:08.845 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:17:08.845 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:17:08.845 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:17:08.845 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:17:08.845 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:17:08.845 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:17:08.845 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:17:08.845 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:17:08.845 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:17:08.845 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:17:08.845 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:17:08.845 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:17:08.845 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:17:08.845 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:17:08.846 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:17:08.846 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:17:08.846 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:17:08.847 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:17:08.847 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:17:08.847 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:17:08.847 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:17:08.847 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:17:08.908 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:17:08.956 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:17:09.029 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:17:09.169 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:17:09.293 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:17:09.462 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:17:09.485 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:17:09.510 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:17:09.510 INFO:teuthology.orchestra.run.smithi017.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:09.510 INFO:teuthology.orchestra.run.smithi017.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:17:09.510 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:17:09.510 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:17:09.510 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:17:09.786 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:17:09.786 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:17:09.786 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-29T11:17:09.787 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:09.787 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:09.787 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:09.787 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:09.787 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:09.787 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:09.787 INFO:teuthology.orchestra.run.smithi094.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:17:09.787 INFO:teuthology.orchestra.run.smithi094.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:17:09.787 INFO:teuthology.orchestra.run.smithi094.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:17:09.787 INFO:teuthology.orchestra.run.smithi094.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:17:09.787 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:17:09.787 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:17:09.787 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:17:09.788 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:17:09.788 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:17:09.788 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:17:09.788 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:17:09.788 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:17:09.788 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:17:09.788 INFO:teuthology.orchestra.run.smithi094.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:17:09.788 INFO:teuthology.orchestra.run.smithi094.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:17:09.788 INFO:teuthology.orchestra.run.smithi094.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:17:09.788 INFO:teuthology.orchestra.run.smithi094.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:17:09.788 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:17:09.788 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:17:09.788 INFO:teuthology.orchestra.run.smithi094.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:17:09.788 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:17:09.789 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:17:09.789 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:17:09.789 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:17:09.789 INFO:teuthology.orchestra.run.smithi094.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:17:09.789 INFO:teuthology.orchestra.run.smithi094.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:17:09.789 INFO:teuthology.orchestra.run.smithi094.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:17:09.789 INFO:teuthology.orchestra.run.smithi094.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:17:09.789 INFO:teuthology.orchestra.run.smithi094.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:17:09.789 INFO:teuthology.orchestra.run.smithi094.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:17:09.789 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:17:09.789 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T11:17:10.127 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-base 2024-03-29T11:17:10.684 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:27 ago on Fri 29 Mar 2024 11:15:43 AM UTC. 2024-03-29T11:17:10.782 INFO:teuthology.orchestra.run.smithi094.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:17:10.833 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T11:17:10.834 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-03-29T11:17:10.834 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T11:17:10.895 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install cephadm 2024-03-29T11:17:11.019 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:17:11.045 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:17:11.046 INFO:teuthology.orchestra.run.smithi017.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:11.046 INFO:teuthology.orchestra.run.smithi017.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:17:11.046 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:17:11.046 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:17:11.046 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:17:11.458 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:28 ago on Fri 29 Mar 2024 11:15:43 AM UTC. 2024-03-29T11:17:11.512 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:17:11.535 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:17:11.535 INFO:teuthology.orchestra.run.smithi017.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:11.535 INFO:teuthology.orchestra.run.smithi017.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:17:11.535 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:17:11.535 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:17:11.535 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:17:11.608 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T11:17:11.609 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T11:17:11.609 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-03-29T11:17:11.609 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T11:17:11.609 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-29T11:17:11.609 INFO:teuthology.orchestra.run.smithi094.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:17:11.609 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:17:11.609 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-29T11:17:11.609 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T11:17:11.610 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-03-29T11:17:11.610 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:17:11.610 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 747 k 2024-03-29T11:17:11.610 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 754 k 2024-03-29T11:17:11.610 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-29T11:17:11.780 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:17:11.802 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:17:11.802 INFO:teuthology.orchestra.run.smithi017.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:11.802 INFO:teuthology.orchestra.run.smithi017.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:17:11.803 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:17:11.803 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:17:11.803 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:17:11.871 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:17:12.022 INFO:teuthology.orchestra.run.smithi094.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.8 MB/s | 747 kB 00:00 2024-03-29T11:17:12.023 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:12.023 INFO:teuthology.orchestra.run.smithi094.stdout:Total 1.8 MB/s | 747 kB 00:00 2024-03-29T11:17:12.023 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-29T11:17:12.028 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-29T11:17:12.028 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-29T11:17:12.033 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-29T11:17:12.034 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-29T11:17:12.110 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-29T11:17:12.538 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:12.618 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:12.962 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:13.252 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:13.252 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:17:13.252 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-29T11:17:13.252 INFO:teuthology.orchestra.run.smithi094.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:13.252 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:17:13.252 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T11:17:13.396 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:17:13.396 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:17:13.396 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:17:13.397 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:17:13.397 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:17:13.397 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:17:13.397 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:17:13.397 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:17:13.397 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:17:13.397 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:17:13.397 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:17:13.397 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:17:13.397 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:17:13.397 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:17:13.397 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:17:13.397 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:17:13.397 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:17:13.398 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:17:13.398 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:17:13.398 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:17:13.398 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:17:13.398 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:17:13.398 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:17:13.398 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:17:13.398 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:17:13.398 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:17:13.398 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:17:13.398 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:17:13.398 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:17:13.398 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:17:13.398 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:17:13.399 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:17:13.399 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:17:13.399 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:17:13.399 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:17:13.399 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:17:13.459 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:17:14.027 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:31 ago on Fri 29 Mar 2024 11:15:43 AM UTC. 2024-03-29T11:17:14.179 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T11:17:14.180 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T11:17:14.180 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-03-29T11:17:14.180 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T11:17:14.181 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-29T11:17:14.181 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:17:14.181 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:17:14.181 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-29T11:17:14.181 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T11:17:14.181 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-03-29T11:17:14.181 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:17:14.181 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 145 k 2024-03-29T11:17:14.181 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 436 k 2024-03-29T11:17:14.181 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-29T11:17:14.344 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:17:14.344 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:17:14.344 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-03-29T11:17:14.344 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:14.344 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:14.344 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:14.344 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:14.344 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:14.345 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:14.345 INFO:teuthology.orchestra.run.smithi017.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:17:14.345 INFO:teuthology.orchestra.run.smithi017.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:17:14.345 INFO:teuthology.orchestra.run.smithi017.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:17:14.345 INFO:teuthology.orchestra.run.smithi017.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:17:14.345 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:17:14.345 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:17:14.345 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:17:14.345 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:17:14.345 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:17:14.345 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:17:14.345 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:17:14.345 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:17:14.345 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:17:14.346 INFO:teuthology.orchestra.run.smithi017.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:17:14.346 INFO:teuthology.orchestra.run.smithi017.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:17:14.346 INFO:teuthology.orchestra.run.smithi017.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:17:14.346 INFO:teuthology.orchestra.run.smithi017.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:17:14.346 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:17:14.346 INFO:teuthology.orchestra.run.smithi017.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:17:14.346 INFO:teuthology.orchestra.run.smithi017.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:17:14.346 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:17:14.346 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:17:14.346 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:17:14.346 INFO:teuthology.orchestra.run.smithi017.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:17:14.346 INFO:teuthology.orchestra.run.smithi017.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:17:14.346 INFO:teuthology.orchestra.run.smithi017.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:17:14.347 INFO:teuthology.orchestra.run.smithi017.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:17:14.347 INFO:teuthology.orchestra.run.smithi017.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:17:14.347 INFO:teuthology.orchestra.run.smithi017.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:17:14.347 INFO:teuthology.orchestra.run.smithi017.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:17:14.347 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:17:14.347 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-03-29T11:17:14.579 INFO:teuthology.orchestra.run.smithi094.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 365 kB/s | 145 kB 00:00 2024-03-29T11:17:14.580 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:14.580 INFO:teuthology.orchestra.run.smithi094.stdout:Total 364 kB/s | 145 kB 00:00 2024-03-29T11:17:14.580 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-29T11:17:14.588 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-29T11:17:14.588 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-29T11:17:14.641 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-29T11:17:14.641 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-29T11:17:14.728 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-base 2024-03-29T11:17:14.784 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-29T11:17:14.855 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:17:14.880 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:17:14.880 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:14.881 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:17:14.881 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:17:15.309 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:15 ago on Fri 29 Mar 2024 11:16:00 AM UTC. 2024-03-29T11:17:15.414 INFO:teuthology.orchestra.run.smithi017.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:17:15.470 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-03-29T11:17:15.471 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2024-03-29T11:17:15.471 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-03-29T11:17:15.536 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install cephadm 2024-03-29T11:17:15.842 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:17:15.842 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:17:15.842 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-29T11:17:15.842 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:15.842 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:17:15.842 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T11:17:16.082 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr 2024-03-29T11:17:16.115 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:16 ago on Fri 29 Mar 2024 11:16:00 AM UTC. 2024-03-29T11:17:16.276 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-03-29T11:17:16.277 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-03-29T11:17:16.277 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repository Size 2024-03-29T11:17:16.277 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-03-29T11:17:16.277 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-03-29T11:17:16.277 INFO:teuthology.orchestra.run.smithi017.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:17:16.277 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:17:16.277 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-03-29T11:17:16.277 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-03-29T11:17:16.277 INFO:teuthology.orchestra.run.smithi017.stdout:Install 1 Package 2024-03-29T11:17:16.278 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:17:16.278 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 747 k 2024-03-29T11:17:16.278 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 754 k 2024-03-29T11:17:16.278 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-03-29T11:17:16.655 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:33 ago on Fri 29 Mar 2024 11:15:43 AM UTC. 2024-03-29T11:17:16.754 INFO:teuthology.orchestra.run.smithi094.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:17:16.806 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T11:17:16.807 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-03-29T11:17:16.807 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T11:17:16.868 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:17:17.014 INFO:teuthology.orchestra.run.smithi017.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.0 MB/s | 747 kB 00:00 2024-03-29T11:17:17.015 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:17.015 INFO:teuthology.orchestra.run.smithi017.stdout:Total 1.0 MB/s | 747 kB 00:00 2024-03-29T11:17:17.015 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-03-29T11:17:17.021 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-03-29T11:17:17.021 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-03-29T11:17:17.027 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-03-29T11:17:17.027 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-03-29T11:17:17.105 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-03-29T11:17:17.429 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:34 ago on Fri 29 Mar 2024 11:15:43 AM UTC. 2024-03-29T11:17:17.489 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:17.562 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:17.579 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T11:17:17.580 INFO:teuthology.orchestra.run.smithi094.stdout:====================================================================================== 2024-03-29T11:17:17.580 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-03-29T11:17:17.580 INFO:teuthology.orchestra.run.smithi094.stdout:====================================================================================== 2024-03-29T11:17:17.580 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-29T11:17:17.580 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:17:17.580 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-03-29T11:17:17.580 INFO:teuthology.orchestra.run.smithi094.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:17:17.580 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:17:17.581 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:17:17.581 INFO:teuthology.orchestra.run.smithi094.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:17:17.581 INFO:teuthology.orchestra.run.smithi094.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:17:17.581 INFO:teuthology.orchestra.run.smithi094.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:17:17.581 INFO:teuthology.orchestra.run.smithi094.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:17:17.581 INFO:teuthology.orchestra.run.smithi094.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:17:17.581 INFO:teuthology.orchestra.run.smithi094.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:17:17.581 INFO:teuthology.orchestra.run.smithi094.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:17:17.581 INFO:teuthology.orchestra.run.smithi094.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:17:17.581 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:17:17.581 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-29T11:17:17.581 INFO:teuthology.orchestra.run.smithi094.stdout:====================================================================================== 2024-03-29T11:17:17.581 INFO:teuthology.orchestra.run.smithi094.stdout:Install 12 Packages 2024-03-29T11:17:17.581 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:17:17.582 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 6.9 M 2024-03-29T11:17:17.582 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 26 M 2024-03-29T11:17:17.582 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-29T11:17:17.917 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:17.934 INFO:teuthology.orchestra.run.smithi094.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 121 kB/s | 31 kB 00:00 2024-03-29T11:17:17.959 INFO:teuthology.orchestra.run.smithi094.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 54 kB/s | 15 kB 00:00 2024-03-29T11:17:18.126 INFO:teuthology.orchestra.run.smithi094.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.6 MB/s | 238 kB 00:00 2024-03-29T11:17:18.160 INFO:teuthology.orchestra.run.smithi094.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.5 MB/s | 1.0 MB 00:00 2024-03-29T11:17:18.211 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:18.212 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:17:18.212 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-03-29T11:17:18.212 INFO:teuthology.orchestra.run.smithi017.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:18.212 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:17:18.212 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-03-29T11:17:18.252 INFO:teuthology.orchestra.run.smithi094.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 3.1 MB/s | 1.8 MB 00:00 2024-03-29T11:17:18.377 INFO:teuthology.orchestra.run.smithi094.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 155 kB/s | 19 kB 00:00 2024-03-29T11:17:18.402 INFO:teuthology.orchestra.run.smithi094.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.1 MB/s | 863 kB 00:00 2024-03-29T11:17:18.421 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:17:18.436 INFO:teuthology.orchestra.run.smithi094.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.9 MB/s | 548 kB 00:00 2024-03-29T11:17:18.486 INFO:teuthology.orchestra.run.smithi094.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 613 kB/s | 31 kB 00:00 2024-03-29T11:17:18.545 INFO:teuthology.orchestra.run.smithi094.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.2 MB/s | 188 kB 00:00 2024-03-29T11:17:18.645 INFO:teuthology.orchestra.run.smithi094.stdout:(11/12): python3-grpcio-tools-1.46.7-10.el9.x86 596 kB/s | 144 kB 00:00 2024-03-29T11:17:18.745 INFO:teuthology.orchestra.run.smithi094.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 5.5 MB/s | 2.0 MB 00:00 2024-03-29T11:17:18.748 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:18.748 INFO:teuthology.orchestra.run.smithi094.stdout:Total 5.9 MB/s | 6.9 MB 00:01 2024-03-29T11:17:18.767 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-29T11:17:18.783 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-29T11:17:18.783 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-29T11:17:18.960 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-29T11:17:18.960 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-29T11:17:18.992 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:19 ago on Fri 29 Mar 2024 11:16:00 AM UTC. 2024-03-29T11:17:19.149 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-03-29T11:17:19.150 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-03-29T11:17:19.150 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repo Size 2024-03-29T11:17:19.150 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-03-29T11:17:19.150 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-03-29T11:17:19.150 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:17:19.150 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:17:19.150 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-03-29T11:17:19.150 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-03-29T11:17:19.150 INFO:teuthology.orchestra.run.smithi017.stdout:Install 1 Package 2024-03-29T11:17:19.150 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:17:19.151 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 145 k 2024-03-29T11:17:19.151 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 436 k 2024-03-29T11:17:19.151 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-03-29T11:17:19.466 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-29T11:17:19.537 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:17:19.572 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:17:19.619 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:17:19.650 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:17:19.695 INFO:teuthology.orchestra.run.smithi017.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 267 kB/s | 145 kB 00:00 2024-03-29T11:17:19.696 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:19.696 INFO:teuthology.orchestra.run.smithi017.stdout:Total 266 kB/s | 145 kB 00:00 2024-03-29T11:17:19.696 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-03-29T11:17:19.704 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-03-29T11:17:19.704 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-03-29T11:17:19.752 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:17:19.759 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-03-29T11:17:19.760 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-03-29T11:17:19.875 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:17:19.905 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-03-29T11:17:19.970 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:17:19.971 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:17:19.996 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:17:19.996 INFO:teuthology.orchestra.run.smithi017.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:19.996 INFO:teuthology.orchestra.run.smithi017.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:17:19.996 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:17:20.021 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:17:20.050 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:17:20.081 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:17:20.323 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:17:20.357 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:17:20.909 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:17:20.909 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:17:20.910 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-03-29T11:17:20.910 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:20.910 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:17:20.910 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-03-29T11:17:21.222 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:17:21.222 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:17:21.222 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:17:21.222 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:17:21.222 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:17:21.222 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:17:21.222 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:17:21.223 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:17:21.223 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:17:21.223 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:17:21.223 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:17:21.223 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:17:21.240 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-mgr 2024-03-29T11:17:21.582 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:17:21.583 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:17:21.583 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-29T11:17:21.583 INFO:teuthology.orchestra.run.smithi094.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:17:21.583 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:21.583 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:21.583 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:21.583 INFO:teuthology.orchestra.run.smithi094.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:17:21.583 INFO:teuthology.orchestra.run.smithi094.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:17:21.583 INFO:teuthology.orchestra.run.smithi094.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:17:21.583 INFO:teuthology.orchestra.run.smithi094.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:17:21.583 INFO:teuthology.orchestra.run.smithi094.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:17:21.583 INFO:teuthology.orchestra.run.smithi094.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:17:21.583 INFO:teuthology.orchestra.run.smithi094.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:17:21.583 INFO:teuthology.orchestra.run.smithi094.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:17:21.583 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:17:21.584 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T11:17:21.825 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:21 ago on Fri 29 Mar 2024 11:16:00 AM UTC. 2024-03-29T11:17:21.840 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:17:21.927 INFO:teuthology.orchestra.run.smithi017.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:17:21.982 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-03-29T11:17:21.984 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2024-03-29T11:17:21.984 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-03-29T11:17:22.046 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:17:22.400 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 11:15:43 AM UTC. 2024-03-29T11:17:22.553 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T11:17:22.555 INFO:teuthology.orchestra.run.smithi094.stdout:============================================================================================ 2024-03-29T11:17:22.555 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-03-29T11:17:22.555 INFO:teuthology.orchestra.run.smithi094.stdout:============================================================================================ 2024-03-29T11:17:22.555 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-29T11:17:22.555 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:17:22.555 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-03-29T11:17:22.555 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:17:22.555 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:17:22.555 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:17:22.555 INFO:teuthology.orchestra.run.smithi094.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:17:22.555 INFO:teuthology.orchestra.run.smithi094.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:17:22.555 INFO:teuthology.orchestra.run.smithi094.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:17:22.555 INFO:teuthology.orchestra.run.smithi094.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:17:22.556 INFO:teuthology.orchestra.run.smithi094.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:17:22.556 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:17:22.556 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:17:22.556 INFO:teuthology.orchestra.run.smithi094.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:17:22.556 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:17:22.556 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-29T11:17:22.556 INFO:teuthology.orchestra.run.smithi094.stdout:============================================================================================ 2024-03-29T11:17:22.556 INFO:teuthology.orchestra.run.smithi094.stdout:Install 12 Packages 2024-03-29T11:17:22.556 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:17:22.557 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 37 M 2024-03-29T11:17:22.557 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 209 M 2024-03-29T11:17:22.557 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-29T11:17:22.614 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:22 ago on Fri 29 Mar 2024 11:16:00 AM UTC. 2024-03-29T11:17:22.770 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-03-29T11:17:22.772 INFO:teuthology.orchestra.run.smithi017.stdout:====================================================================================== 2024-03-29T11:17:22.772 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repository Size 2024-03-29T11:17:22.772 INFO:teuthology.orchestra.run.smithi017.stdout:====================================================================================== 2024-03-29T11:17:22.772 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-03-29T11:17:22.772 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:17:22.772 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2024-03-29T11:17:22.772 INFO:teuthology.orchestra.run.smithi017.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:17:22.772 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:17:22.772 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:17:22.772 INFO:teuthology.orchestra.run.smithi017.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:17:22.772 INFO:teuthology.orchestra.run.smithi017.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:17:22.772 INFO:teuthology.orchestra.run.smithi017.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:17:22.773 INFO:teuthology.orchestra.run.smithi017.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:17:22.773 INFO:teuthology.orchestra.run.smithi017.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:17:22.773 INFO:teuthology.orchestra.run.smithi017.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:17:22.773 INFO:teuthology.orchestra.run.smithi017.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:17:22.773 INFO:teuthology.orchestra.run.smithi017.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:17:22.773 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:17:22.773 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-03-29T11:17:22.773 INFO:teuthology.orchestra.run.smithi017.stdout:====================================================================================== 2024-03-29T11:17:22.773 INFO:teuthology.orchestra.run.smithi017.stdout:Install 12 Packages 2024-03-29T11:17:22.773 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:17:22.774 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 6.9 M 2024-03-29T11:17:22.774 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 26 M 2024-03-29T11:17:22.774 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-03-29T11:17:22.861 INFO:teuthology.orchestra.run.smithi094.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 248 kB/s | 32 kB 00:00 2024-03-29T11:17:22.895 INFO:teuthology.orchestra.run.smithi094.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 515 kB/s | 17 kB 00:00 2024-03-29T11:17:22.929 INFO:teuthology.orchestra.run.smithi094.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-29T11:17:23.096 INFO:teuthology.orchestra.run.smithi094.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 28 MB/s | 4.7 MB 00:00 2024-03-29T11:17:23.133 INFO:teuthology.orchestra.run.smithi017.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 121 kB/s | 31 kB 00:00 2024-03-29T11:17:23.175 INFO:teuthology.orchestra.run.smithi017.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 51 kB/s | 15 kB 00:00 2024-03-29T11:17:23.229 INFO:teuthology.orchestra.run.smithi094.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-29T11:17:23.271 INFO:teuthology.orchestra.run.smithi094.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.3 MB/s | 217 kB 00:00 2024-03-29T11:17:23.367 INFO:teuthology.orchestra.run.smithi017.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.4 MB/s | 238 kB 00:00 2024-03-29T11:17:23.380 INFO:teuthology.orchestra.run.smithi094.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.6 MB/s | 3.0 MB 00:00 2024-03-29T11:17:23.592 INFO:teuthology.orchestra.run.smithi017.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.2 MB/s | 1.0 MB 00:00 2024-03-29T11:17:23.626 INFO:teuthology.orchestra.run.smithi017.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 2.4 MB/s | 1.8 MB 00:00 2024-03-29T11:17:23.772 INFO:teuthology.orchestra.run.smithi094.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 31 MB/s | 16 MB 00:00 2024-03-29T11:17:23.784 INFO:teuthology.orchestra.run.smithi017.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.0 MB/s | 863 kB 00:00 2024-03-29T11:17:23.989 INFO:teuthology.orchestra.run.smithi094.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 903 kB/s | 195 kB 00:00 2024-03-29T11:17:23.993 INFO:teuthology.orchestra.run.smithi017.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 53 kB/s | 19 kB 00:00 2024-03-29T11:17:24.189 INFO:teuthology.orchestra.run.smithi094.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.2 MB/s | 245 kB 00:00 2024-03-29T11:17:24.285 INFO:teuthology.orchestra.run.smithi017.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 495 kB/s | 144 kB 00:00 2024-03-29T11:17:24.310 INFO:teuthology.orchestra.run.smithi017.stdout:(9/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 764 kB/s | 548 kB 00:00 2024-03-29T11:17:24.394 INFO:teuthology.orchestra.run.smithi017.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 284 kB/s | 31 kB 00:00 2024-03-29T11:17:24.427 INFO:teuthology.orchestra.run.smithi017.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.6 MB/s | 188 kB 00:00 2024-03-29T11:17:24.628 INFO:teuthology.orchestra.run.smithi017.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.4 MB/s | 2.0 MB 00:00 2024-03-29T11:17:24.633 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:24.633 INFO:teuthology.orchestra.run.smithi017.stdout:Total 3.7 MB/s | 6.9 MB 00:01 2024-03-29T11:17:24.653 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-03-29T11:17:24.672 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-03-29T11:17:24.672 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-03-29T11:17:24.859 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-03-29T11:17:24.860 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-03-29T11:17:24.939 INFO:teuthology.orchestra.run.smithi094.stdout:(11/12): libgfortran-11.4.1-3.el9.x86_64.rpm 516 kB/s | 803 kB 00:01 2024-03-29T11:17:25.383 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-03-29T11:17:25.463 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:17:25.505 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:17:25.553 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:17:25.591 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:17:25.700 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:17:25.830 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:17:25.922 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:17:25.972 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:17:25.999 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:17:26.031 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:17:26.279 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:17:26.315 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:17:27.087 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:17:27.088 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:17:27.088 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:17:27.088 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:17:27.088 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:17:27.088 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:17:27.088 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:17:27.088 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:17:27.088 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:17:27.088 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:17:27.088 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:17:27.088 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:17:27.438 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:17:27.438 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:17:27.438 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-03-29T11:17:27.438 INFO:teuthology.orchestra.run.smithi017.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:17:27.438 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:27.438 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:27.439 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:27.439 INFO:teuthology.orchestra.run.smithi017.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:17:27.439 INFO:teuthology.orchestra.run.smithi017.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:17:27.439 INFO:teuthology.orchestra.run.smithi017.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:17:27.439 INFO:teuthology.orchestra.run.smithi017.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:17:27.439 INFO:teuthology.orchestra.run.smithi017.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:17:27.439 INFO:teuthology.orchestra.run.smithi017.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:17:27.439 INFO:teuthology.orchestra.run.smithi017.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:17:27.439 INFO:teuthology.orchestra.run.smithi017.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:17:27.439 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:17:27.439 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-03-29T11:17:27.861 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:17:28.434 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:28 ago on Fri 29 Mar 2024 11:16:00 AM UTC. 2024-03-29T11:17:28.596 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-03-29T11:17:28.597 INFO:teuthology.orchestra.run.smithi017.stdout:============================================================================================ 2024-03-29T11:17:28.597 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repository Size 2024-03-29T11:17:28.597 INFO:teuthology.orchestra.run.smithi017.stdout:============================================================================================ 2024-03-29T11:17:28.597 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-03-29T11:17:28.597 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:17:28.597 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2024-03-29T11:17:28.597 INFO:teuthology.orchestra.run.smithi017.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:17:28.598 INFO:teuthology.orchestra.run.smithi017.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:17:28.598 INFO:teuthology.orchestra.run.smithi017.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:17:28.598 INFO:teuthology.orchestra.run.smithi017.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:17:28.598 INFO:teuthology.orchestra.run.smithi017.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:17:28.598 INFO:teuthology.orchestra.run.smithi017.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:17:28.598 INFO:teuthology.orchestra.run.smithi017.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:17:28.598 INFO:teuthology.orchestra.run.smithi017.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:17:28.598 INFO:teuthology.orchestra.run.smithi017.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:17:28.598 INFO:teuthology.orchestra.run.smithi017.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:17:28.598 INFO:teuthology.orchestra.run.smithi017.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:17:28.598 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:17:28.598 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-03-29T11:17:28.598 INFO:teuthology.orchestra.run.smithi017.stdout:============================================================================================ 2024-03-29T11:17:28.598 INFO:teuthology.orchestra.run.smithi017.stdout:Install 12 Packages 2024-03-29T11:17:28.599 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:17:28.599 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 37 M 2024-03-29T11:17:28.599 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 209 M 2024-03-29T11:17:28.599 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-03-29T11:17:28.935 INFO:teuthology.orchestra.run.smithi017.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 232 kB/s | 32 kB 00:00 2024-03-29T11:17:28.977 INFO:teuthology.orchestra.run.smithi017.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 412 kB/s | 17 kB 00:00 2024-03-29T11:17:29.019 INFO:teuthology.orchestra.run.smithi017.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 837 kB/s | 34 kB 00:00 2024-03-29T11:17:29.131 INFO:teuthology.orchestra.run.smithi094.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 1.2 MB/s | 7.4 MB 00:06 2024-03-29T11:17:29.133 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:29.133 INFO:teuthology.orchestra.run.smithi094.stdout:Total 5.6 MB/s | 37 MB 00:06 2024-03-29T11:17:29.152 INFO:teuthology.orchestra.run.smithi017.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 8.4 MB/s | 3.0 MB 00:00 2024-03-29T11:17:29.176 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-29T11:17:29.202 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-29T11:17:29.203 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-29T11:17:29.564 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-29T11:17:29.564 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-29T11:17:30.119 INFO:teuthology.orchestra.run.smithi017.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 4.3 MB/s | 4.7 MB 00:01 2024-03-29T11:17:30.211 INFO:teuthology.orchestra.run.smithi017.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.5 MB/s | 4.8 MB 00:01 2024-03-29T11:17:30.236 INFO:teuthology.orchestra.run.smithi017.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.8 MB/s | 217 kB 00:00 2024-03-29T11:17:30.256 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-29T11:17:30.328 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:17:30.360 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:17:30.459 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:17:30.497 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:17:30.620 INFO:teuthology.orchestra.run.smithi017.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2024-03-29T11:17:30.673 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:17:30.704 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:17:30.786 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:17:31.137 INFO:teuthology.orchestra.run.smithi017.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 893 kB/s | 803 kB 00:00 2024-03-29T11:17:31.162 INFO:teuthology.orchestra.run.smithi017.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 360 kB/s | 195 kB 00:00 2024-03-29T11:17:31.362 INFO:teuthology.orchestra.run.smithi017.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.1 MB/s | 245 kB 00:00 2024-03-29T11:17:31.468 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:17:31.587 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:17:32.863 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:17:33.854 INFO:teuthology.orchestra.run.smithi017.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 1.5 MB/s | 7.4 MB 00:05 2024-03-29T11:17:33.857 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:33.857 INFO:teuthology.orchestra.run.smithi017.stdout:Total 7.0 MB/s | 37 MB 00:05 2024-03-29T11:17:33.873 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:17:33.902 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-03-29T11:17:33.905 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:17:33.929 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-03-29T11:17:33.929 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-03-29T11:17:34.301 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-03-29T11:17:34.301 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-03-29T11:17:35.029 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-03-29T11:17:35.094 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:17:35.125 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:17:35.223 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:17:35.262 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:17:35.446 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:17:35.477 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:17:35.561 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:17:36.243 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:17:36.352 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:17:36.423 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:17:36.423 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:17:36.423 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:17:36.423 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:17:36.423 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:17:36.423 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:17:36.423 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:17:36.423 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:17:36.423 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:17:36.423 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:17:36.423 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:17:36.423 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:17:36.866 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:17:36.866 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:17:36.866 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-29T11:17:36.866 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:36.866 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:17:36.866 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:17:36.866 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:17:36.866 INFO:teuthology.orchestra.run.smithi094.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:17:36.866 INFO:teuthology.orchestra.run.smithi094.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:17:36.867 INFO:teuthology.orchestra.run.smithi094.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:17:36.867 INFO:teuthology.orchestra.run.smithi094.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:17:36.867 INFO:teuthology.orchestra.run.smithi094.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:17:36.867 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:17:36.867 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:17:36.867 INFO:teuthology.orchestra.run.smithi094.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:17:36.867 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:17:36.867 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T11:17:37.351 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:17:37.659 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:17:37.937 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:54 ago on Fri 29 Mar 2024 11:15:43 AM UTC. 2024-03-29T11:17:38.104 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T11:17:38.105 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================== 2024-03-29T11:17:38.105 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-03-29T11:17:38.105 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================== 2024-03-29T11:17:38.105 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-29T11:17:38.105 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:17:38.105 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-03-29T11:17:38.106 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:17:38.106 INFO:teuthology.orchestra.run.smithi094.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:17:38.106 INFO:teuthology.orchestra.run.smithi094.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:17:38.106 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:17:38.106 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:17:38.106 INFO:teuthology.orchestra.run.smithi094.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:17:38.106 INFO:teuthology.orchestra.run.smithi094.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:17:38.106 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:17:38.106 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:17:38.106 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:17:38.106 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:17:38.106 INFO:teuthology.orchestra.run.smithi094.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:17:38.106 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:17:38.107 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-29T11:17:38.107 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================== 2024-03-29T11:17:38.107 INFO:teuthology.orchestra.run.smithi094.stdout:Install 13 Packages 2024-03-29T11:17:38.107 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:17:38.107 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 2.0 M 2024-03-29T11:17:38.107 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 26 M 2024-03-29T11:17:38.108 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-29T11:17:38.417 INFO:teuthology.orchestra.run.smithi094.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 215 kB/s | 24 kB 00:00 2024-03-29T11:17:38.467 INFO:teuthology.orchestra.run.smithi094.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 110 kB/s | 18 kB 00:00 2024-03-29T11:17:38.509 INFO:teuthology.orchestra.run.smithi094.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.5 MB/s | 174 kB 00:00 2024-03-29T11:17:38.534 INFO:teuthology.orchestra.run.smithi094.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-29T11:17:38.559 INFO:teuthology.orchestra.run.smithi094.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.2 MB/s | 211 kB 00:00 2024-03-29T11:17:38.584 INFO:teuthology.orchestra.run.smithi094.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 856 kB/s | 43 kB 00:00 2024-03-29T11:17:38.735 INFO:teuthology.orchestra.run.smithi094.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 184 kB/s | 32 kB 00:00 2024-03-29T11:17:38.736 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:17:38.760 INFO:teuthology.orchestra.run.smithi094.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 85 kB/s | 15 kB 00:00 2024-03-29T11:17:38.768 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:17:38.811 INFO:teuthology.orchestra.run.smithi094.stdout:(9/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 103 kB/s | 52 kB 00:00 2024-03-29T11:17:38.836 INFO:teuthology.orchestra.run.smithi094.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 2.0 MB/s | 203 kB 00:00 2024-03-29T11:17:38.894 INFO:teuthology.orchestra.run.smithi094.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 7.6 MB/s | 1.0 MB 00:00 2024-03-29T11:17:38.920 INFO:teuthology.orchestra.run.smithi094.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 544 kB/s | 59 kB 00:00 2024-03-29T11:17:38.945 INFO:teuthology.orchestra.run.smithi094.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 827 kB/s | 90 kB 00:00 2024-03-29T11:17:38.948 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:38.949 INFO:teuthology.orchestra.run.smithi094.stdout:Total 2.4 MB/s | 2.0 MB 00:00 2024-03-29T11:17:38.949 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-29T11:17:38.962 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-29T11:17:38.962 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-29T11:17:39.052 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-29T11:17:39.052 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-29T11:17:39.568 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-29T11:17:39.733 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:17:39.779 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:17:39.843 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:17:39.881 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:17:39.921 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:17:40.014 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:17:40.123 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:17:40.182 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:17:40.894 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:17:40.945 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:17:40.979 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:17:41.027 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:17:41.049 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:17:41.216 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:17:41.216 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:17:41.216 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:17:41.216 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:17:41.216 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:17:41.217 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:17:41.217 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:17:41.217 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:17:41.217 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:17:41.217 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:17:41.217 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:17:41.217 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:17:41.645 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:17:41.645 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:17:41.646 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-03-29T11:17:41.646 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:41.646 INFO:teuthology.orchestra.run.smithi017.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:17:41.646 INFO:teuthology.orchestra.run.smithi017.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:17:41.646 INFO:teuthology.orchestra.run.smithi017.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:17:41.646 INFO:teuthology.orchestra.run.smithi017.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:17:41.646 INFO:teuthology.orchestra.run.smithi017.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:17:41.646 INFO:teuthology.orchestra.run.smithi017.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:17:41.646 INFO:teuthology.orchestra.run.smithi017.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:17:41.646 INFO:teuthology.orchestra.run.smithi017.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:17:41.646 INFO:teuthology.orchestra.run.smithi017.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:17:41.646 INFO:teuthology.orchestra.run.smithi017.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:17:41.646 INFO:teuthology.orchestra.run.smithi017.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:17:41.647 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:17:41.647 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-03-29T11:17:41.852 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:17:41.852 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:17:41.852 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:17:41.852 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:17:41.852 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:17:41.852 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:17:41.852 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:17:41.853 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:17:41.853 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:17:41.853 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:17:41.853 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:17:41.853 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:17:41.853 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:17:42.058 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:17:42.223 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:17:42.223 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:17:42.223 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-29T11:17:42.223 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:42.223 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:17:42.223 INFO:teuthology.orchestra.run.smithi094.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:17:42.223 INFO:teuthology.orchestra.run.smithi094.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:17:42.223 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:17:42.223 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:17:42.223 INFO:teuthology.orchestra.run.smithi094.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:17:42.223 INFO:teuthology.orchestra.run.smithi094.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:17:42.224 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:17:42.224 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:17:42.224 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:17:42.224 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:17:42.224 INFO:teuthology.orchestra.run.smithi094.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:17:42.224 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:17:42.224 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T11:17:42.470 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:17:42.632 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 11:16:00 AM UTC. 2024-03-29T11:17:42.791 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-03-29T11:17:42.792 INFO:teuthology.orchestra.run.smithi017.stdout:======================================================================================== 2024-03-29T11:17:42.793 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repository Size 2024-03-29T11:17:42.793 INFO:teuthology.orchestra.run.smithi017.stdout:======================================================================================== 2024-03-29T11:17:42.793 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-03-29T11:17:42.793 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:17:42.793 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2024-03-29T11:17:42.793 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:17:42.793 INFO:teuthology.orchestra.run.smithi017.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:17:42.793 INFO:teuthology.orchestra.run.smithi017.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:17:42.793 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:17:42.793 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:17:42.793 INFO:teuthology.orchestra.run.smithi017.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:17:42.793 INFO:teuthology.orchestra.run.smithi017.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:17:42.793 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:17:42.793 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:17:42.793 INFO:teuthology.orchestra.run.smithi017.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:17:42.794 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:17:42.794 INFO:teuthology.orchestra.run.smithi017.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:17:42.794 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:17:42.794 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-03-29T11:17:42.794 INFO:teuthology.orchestra.run.smithi017.stdout:======================================================================================== 2024-03-29T11:17:42.794 INFO:teuthology.orchestra.run.smithi017.stdout:Install 13 Packages 2024-03-29T11:17:42.794 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:17:42.795 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 2.0 M 2024-03-29T11:17:42.795 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 26 M 2024-03-29T11:17:42.795 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-03-29T11:17:42.977 INFO:teuthology.orchestra.run.smithi017.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 185 kB/s | 24 kB 00:00 2024-03-29T11:17:43.002 INFO:teuthology.orchestra.run.smithi017.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 115 kB/s | 18 kB 00:00 2024-03-29T11:17:43.038 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:00 ago on Fri 29 Mar 2024 11:15:43 AM UTC. 2024-03-29T11:17:43.044 INFO:teuthology.orchestra.run.smithi017.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-03-29T11:17:43.069 INFO:teuthology.orchestra.run.smithi017.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-29T11:17:43.094 INFO:teuthology.orchestra.run.smithi017.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T11:17:43.120 INFO:teuthology.orchestra.run.smithi017.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 859 kB/s | 43 kB 00:00 2024-03-29T11:17:43.192 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T11:17:43.193 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================== 2024-03-29T11:17:43.193 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-03-29T11:17:43.193 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================== 2024-03-29T11:17:43.193 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-29T11:17:43.193 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:17:43.193 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-03-29T11:17:43.193 INFO:teuthology.orchestra.run.smithi094.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:17:43.193 INFO:teuthology.orchestra.run.smithi094.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:17:43.194 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:17:43.194 INFO:teuthology.orchestra.run.smithi094.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:17:43.194 INFO:teuthology.orchestra.run.smithi094.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:17:43.194 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:17:43.194 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-29T11:17:43.194 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================== 2024-03-29T11:17:43.194 INFO:teuthology.orchestra.run.smithi094.stdout:Install 6 Packages 2024-03-29T11:17:43.194 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:17:43.195 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 6.8 M 2024-03-29T11:17:43.195 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 33 M 2024-03-29T11:17:43.195 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-29T11:17:43.353 INFO:teuthology.orchestra.run.smithi017.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 103 kB/s | 52 kB 00:00 2024-03-29T11:17:43.379 INFO:teuthology.orchestra.run.smithi017.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 58 kB/s | 15 kB 00:00 2024-03-29T11:17:43.404 INFO:teuthology.orchestra.run.smithi017.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 104 kB/s | 32 kB 00:00 2024-03-29T11:17:43.546 INFO:teuthology.orchestra.run.smithi017.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 416 kB/s | 59 kB 00:00 2024-03-29T11:17:43.572 INFO:teuthology.orchestra.run.smithi017.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 930 kB/s | 203 kB 00:00 2024-03-29T11:17:43.663 INFO:teuthology.orchestra.run.smithi017.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 767 kB/s | 90 kB 00:00 2024-03-29T11:17:43.839 INFO:teuthology.orchestra.run.smithi094.stdout:(1/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 317 kB/s | 138 kB 00:00 2024-03-29T11:17:43.897 INFO:teuthology.orchestra.run.smithi017.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.0 MB/s | 1.0 MB 00:00 2024-03-29T11:17:43.901 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:43.901 INFO:teuthology.orchestra.run.smithi017.stdout:Total 1.9 MB/s | 2.0 MB 00:01 2024-03-29T11:17:43.902 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-03-29T11:17:43.916 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-03-29T11:17:43.916 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-03-29T11:17:44.017 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-03-29T11:17:44.018 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-03-29T11:17:44.031 INFO:teuthology.orchestra.run.smithi094.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 9.2 MB/s | 5.8 MB 00:00 2024-03-29T11:17:44.265 INFO:teuthology.orchestra.run.smithi094.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 247 kB/s | 58 kB 00:00 2024-03-29T11:17:44.290 INFO:teuthology.orchestra.run.smithi094.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.2 MB/s | 548 kB 00:00 2024-03-29T11:17:44.340 INFO:teuthology.orchestra.run.smithi094.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 678 kB/s | 51 kB 00:00 2024-03-29T11:17:44.390 INFO:teuthology.orchestra.run.smithi094.stdout:(6/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 251 kB/s | 248 kB 00:00 2024-03-29T11:17:44.392 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:44.392 INFO:teuthology.orchestra.run.smithi094.stdout:Total 5.7 MB/s | 6.8 MB 00:01 2024-03-29T11:17:44.405 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-29T11:17:44.415 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-29T11:17:44.416 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-29T11:17:44.505 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-29T11:17:44.505 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-29T11:17:44.547 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-03-29T11:17:44.726 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:17:44.782 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:17:44.798 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-29T11:17:44.838 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:17:44.874 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:17:44.916 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:17:44.962 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:17:45.012 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:17:45.017 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:17:45.121 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:17:45.185 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:17:45.527 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:17:45.612 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:17:45.673 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:17:45.696 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:17:45.961 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:17:46.006 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:17:46.040 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:17:46.089 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:17:46.118 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:17:46.522 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:17:46.522 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:17:46.522 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:17:46.522 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:17:46.523 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:17:46.523 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:17:46.829 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:17:46.829 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:17:46.829 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-29T11:17:46.829 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:46.829 INFO:teuthology.orchestra.run.smithi094.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:17:46.829 INFO:teuthology.orchestra.run.smithi094.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:17:46.829 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:17:46.829 INFO:teuthology.orchestra.run.smithi094.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:17:46.830 INFO:teuthology.orchestra.run.smithi094.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:17:46.830 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:17:46.830 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T11:17:46.935 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:17:46.935 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:17:46.935 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:17:46.935 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:17:46.936 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:17:46.936 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:17:46.936 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:17:46.936 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:17:46.936 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:17:46.936 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:17:46.936 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:17:46.936 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:17:46.936 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:17:47.066 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-fuse 2024-03-29T11:17:47.294 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:17:47.295 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:17:47.295 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-03-29T11:17:47.295 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:47.295 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:17:47.295 INFO:teuthology.orchestra.run.smithi017.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:17:47.295 INFO:teuthology.orchestra.run.smithi017.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:17:47.295 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:17:47.295 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:17:47.295 INFO:teuthology.orchestra.run.smithi017.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:17:47.295 INFO:teuthology.orchestra.run.smithi017.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:17:47.295 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:17:47.295 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:17:47.295 INFO:teuthology.orchestra.run.smithi017.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:17:47.296 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:17:47.296 INFO:teuthology.orchestra.run.smithi017.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:17:47.296 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:17:47.296 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-03-29T11:17:47.543 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:17:47.625 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:04 ago on Fri 29 Mar 2024 11:15:43 AM UTC. 2024-03-29T11:17:47.774 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T11:17:47.774 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T11:17:47.774 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-03-29T11:17:47.775 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T11:17:47.775 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-29T11:17:47.775 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:17:47.775 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-03-29T11:17:47.775 INFO:teuthology.orchestra.run.smithi094.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:17:47.775 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:17:47.775 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-29T11:17:47.775 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T11:17:47.775 INFO:teuthology.orchestra.run.smithi094.stdout:Install 2 Packages 2024-03-29T11:17:47.775 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:17:47.776 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 964 k 2024-03-29T11:17:47.776 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 2.8 M 2024-03-29T11:17:47.776 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-29T11:17:47.893 INFO:teuthology.orchestra.run.smithi094.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 674 kB/s | 79 kB 00:00 2024-03-29T11:17:48.121 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:48 ago on Fri 29 Mar 2024 11:16:00 AM UTC. 2024-03-29T11:17:48.281 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-03-29T11:17:48.282 INFO:teuthology.orchestra.run.smithi017.stdout:======================================================================================== 2024-03-29T11:17:48.282 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repository Size 2024-03-29T11:17:48.282 INFO:teuthology.orchestra.run.smithi017.stdout:======================================================================================== 2024-03-29T11:17:48.282 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-03-29T11:17:48.282 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:17:48.282 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2024-03-29T11:17:48.282 INFO:teuthology.orchestra.run.smithi017.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:17:48.282 INFO:teuthology.orchestra.run.smithi017.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:17:48.283 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:17:48.283 INFO:teuthology.orchestra.run.smithi017.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:17:48.283 INFO:teuthology.orchestra.run.smithi017.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:17:48.283 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:17:48.283 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-03-29T11:17:48.283 INFO:teuthology.orchestra.run.smithi017.stdout:======================================================================================== 2024-03-29T11:17:48.283 INFO:teuthology.orchestra.run.smithi017.stdout:Install 6 Packages 2024-03-29T11:17:48.283 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:17:48.283 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 6.8 M 2024-03-29T11:17:48.283 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 33 M 2024-03-29T11:17:48.284 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-03-29T11:17:48.468 INFO:teuthology.orchestra.run.smithi094.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 1.2 MB/s | 885 kB 00:00 2024-03-29T11:17:48.469 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:48.470 INFO:teuthology.orchestra.run.smithi094.stdout:Total 1.4 MB/s | 964 kB 00:00 2024-03-29T11:17:48.470 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-29T11:17:48.490 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-29T11:17:48.490 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-29T11:17:48.564 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-29T11:17:48.564 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-29T11:17:48.683 INFO:teuthology.orchestra.run.smithi017.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-03-29T11:17:48.747 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-29T11:17:48.758 INFO:teuthology.orchestra.run.smithi017.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-29T11:17:48.887 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:17:48.953 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:17:49.050 INFO:teuthology.orchestra.run.smithi017.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 243 kB/s | 138 kB 00:00 2024-03-29T11:17:49.076 INFO:teuthology.orchestra.run.smithi017.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 182 kB/s | 58 kB 00:00 2024-03-29T11:17:49.142 INFO:teuthology.orchestra.run.smithi017.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 557 kB/s | 51 kB 00:00 2024-03-29T11:17:49.184 INFO:teuthology.orchestra.run.smithi017.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.1 MB/s | 548 kB 00:00 2024-03-29T11:17:49.187 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:49.188 INFO:teuthology.orchestra.run.smithi017.stdout:Total 7.5 MB/s | 6.8 MB 00:00 2024-03-29T11:17:49.201 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-03-29T11:17:49.211 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-03-29T11:17:49.211 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-03-29T11:17:49.304 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-03-29T11:17:49.304 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-03-29T11:17:49.603 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:17:49.604 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:17:49.605 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-03-29T11:17:49.780 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:17:49.829 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:17:49.929 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:17:49.929 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:17:49.929 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-29T11:17:49.929 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:17:49.929 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:17:49.929 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T11:17:50.223 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-volume 2024-03-29T11:17:50.379 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:17:50.465 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:17:50.528 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:17:50.557 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:17:50.797 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:07 ago on Fri 29 Mar 2024 11:15:43 AM UTC. 2024-03-29T11:17:50.948 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T11:17:50.949 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T11:17:50.949 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-03-29T11:17:50.949 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T11:17:50.949 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-29T11:17:50.949 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:17:50.950 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-03-29T11:17:50.950 INFO:teuthology.orchestra.run.smithi094.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:17:50.950 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:17:50.950 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-29T11:17:50.950 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T11:17:50.950 INFO:teuthology.orchestra.run.smithi094.stdout:Install 2 Packages 2024-03-29T11:17:50.950 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:17:50.950 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 332 k 2024-03-29T11:17:50.950 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 1.5 M 2024-03-29T11:17:50.950 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-29T11:17:51.132 INFO:teuthology.orchestra.run.smithi094.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 378 kB/s | 69 kB 00:00 2024-03-29T11:17:51.365 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:17:51.365 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:17:51.365 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:17:51.365 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:17:51.366 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:17:51.366 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:17:51.524 INFO:teuthology.orchestra.run.smithi094.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 460 kB/s | 264 kB 00:00 2024-03-29T11:17:51.525 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:51.525 INFO:teuthology.orchestra.run.smithi094.stdout:Total 577 kB/s | 332 kB 00:00 2024-03-29T11:17:51.525 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-29T11:17:51.534 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-29T11:17:51.534 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-29T11:17:51.574 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-29T11:17:51.574 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-29T11:17:51.676 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:17:51.677 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:17:51.677 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-03-29T11:17:51.677 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:51.677 INFO:teuthology.orchestra.run.smithi017.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:17:51.677 INFO:teuthology.orchestra.run.smithi017.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:17:51.677 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:17:51.677 INFO:teuthology.orchestra.run.smithi017.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:17:51.677 INFO:teuthology.orchestra.run.smithi017.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:17:51.677 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:17:51.677 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-03-29T11:17:51.724 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-29T11:17:51.915 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:17:51.919 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-fuse 2024-03-29T11:17:51.943 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:17:51.960 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:17:51.960 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:51.960 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:17:51.960 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:17:52.494 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:52 ago on Fri 29 Mar 2024 11:16:00 AM UTC. 2024-03-29T11:17:52.636 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:17:52.651 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-03-29T11:17:52.651 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-03-29T11:17:52.651 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repository Size 2024-03-29T11:17:52.652 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-03-29T11:17:52.652 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-03-29T11:17:52.652 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:17:52.652 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2024-03-29T11:17:52.652 INFO:teuthology.orchestra.run.smithi017.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:17:52.652 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:17:52.652 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-03-29T11:17:52.652 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-03-29T11:17:52.652 INFO:teuthology.orchestra.run.smithi017.stdout:Install 2 Packages 2024-03-29T11:17:52.652 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:17:52.653 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 964 k 2024-03-29T11:17:52.653 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 2.8 M 2024-03-29T11:17:52.653 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-03-29T11:17:52.769 INFO:teuthology.orchestra.run.smithi017.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 678 kB/s | 79 kB 00:00 2024-03-29T11:17:52.970 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:17:52.970 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:17:52.970 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-29T11:17:52.970 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:52.970 INFO:teuthology.orchestra.run.smithi094.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:17:52.970 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:17:52.970 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T11:17:53.314 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install librados-devel 2024-03-29T11:17:53.845 INFO:teuthology.orchestra.run.smithi017.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 743 kB/s | 885 kB 00:01 2024-03-29T11:17:53.846 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:53.846 INFO:teuthology.orchestra.run.smithi017.stdout:Total 807 kB/s | 964 kB 00:01 2024-03-29T11:17:53.847 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-03-29T11:17:53.872 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-03-29T11:17:53.872 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-03-29T11:17:53.889 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:10 ago on Fri 29 Mar 2024 11:15:43 AM UTC. 2024-03-29T11:17:53.954 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-03-29T11:17:53.954 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-03-29T11:17:54.039 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T11:17:54.040 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T11:17:54.040 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-03-29T11:17:54.040 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T11:17:54.040 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-29T11:17:54.040 INFO:teuthology.orchestra.run.smithi094.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:17:54.040 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:17:54.040 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-29T11:17:54.040 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T11:17:54.040 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-03-29T11:17:54.041 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:17:54.041 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 127 k 2024-03-29T11:17:54.041 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 456 k 2024-03-29T11:17:54.041 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-29T11:17:54.140 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-03-29T11:17:54.277 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:17:54.349 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:17:54.995 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:17:54.995 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:17:55.299 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:17:55.299 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:17:55.299 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-03-29T11:17:55.299 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:17:55.299 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:17:55.300 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-03-29T11:17:55.569 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-volume 2024-03-29T11:17:56.149 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:56 ago on Fri 29 Mar 2024 11:16:00 AM UTC. 2024-03-29T11:17:56.307 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-03-29T11:17:56.308 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-03-29T11:17:56.308 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repository Size 2024-03-29T11:17:56.308 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-03-29T11:17:56.308 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-03-29T11:17:56.308 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:17:56.308 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2024-03-29T11:17:56.308 INFO:teuthology.orchestra.run.smithi017.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:17:56.308 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:17:56.308 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-03-29T11:17:56.308 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-03-29T11:17:56.308 INFO:teuthology.orchestra.run.smithi017.stdout:Install 2 Packages 2024-03-29T11:17:56.309 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:17:56.309 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 332 k 2024-03-29T11:17:56.309 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 1.5 M 2024-03-29T11:17:56.309 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-03-29T11:17:56.434 INFO:teuthology.orchestra.run.smithi017.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 550 kB/s | 69 kB 00:00 2024-03-29T11:17:56.447 INFO:teuthology.orchestra.run.smithi094.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 53 kB/s | 127 kB 00:02 2024-03-29T11:17:56.447 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:56.448 INFO:teuthology.orchestra.run.smithi094.stdout:Total 53 kB/s | 127 kB 00:02 2024-03-29T11:17:56.448 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-29T11:17:56.455 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-29T11:17:56.455 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-29T11:17:56.515 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-29T11:17:56.516 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-29T11:17:56.672 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-29T11:17:56.676 INFO:teuthology.orchestra.run.smithi017.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 719 kB/s | 264 kB 00:00 2024-03-29T11:17:56.678 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:56.678 INFO:teuthology.orchestra.run.smithi017.stdout:Total 900 kB/s | 332 kB 00:00 2024-03-29T11:17:56.679 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-03-29T11:17:56.687 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-03-29T11:17:56.688 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-03-29T11:17:56.727 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-03-29T11:17:56.728 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-03-29T11:17:56.804 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:17:56.882 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-03-29T11:17:57.073 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:17:57.112 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:17:57.127 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:17:57.127 INFO:teuthology.orchestra.run.smithi017.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:57.127 INFO:teuthology.orchestra.run.smithi017.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:17:57.128 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:17:57.254 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:17:57.545 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:17:57.545 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:17:57.545 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-29T11:17:57.545 INFO:teuthology.orchestra.run.smithi094.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:57.545 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:17:57.545 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T11:17:57.808 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install libcephfs2 2024-03-29T11:17:57.854 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:17:58.173 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:17:58.173 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:17:58.173 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-03-29T11:17:58.173 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:58.173 INFO:teuthology.orchestra.run.smithi017.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:17:58.173 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:17:58.174 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-03-29T11:17:58.385 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:15 ago on Fri 29 Mar 2024 11:15:43 AM UTC. 2024-03-29T11:17:58.486 INFO:teuthology.orchestra.run.smithi094.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:17:58.516 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install librados-devel 2024-03-29T11:17:58.539 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T11:17:58.540 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-03-29T11:17:58.540 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T11:17:58.613 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install libcephfs-devel 2024-03-29T11:17:59.097 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:59 ago on Fri 29 Mar 2024 11:16:00 AM UTC. 2024-03-29T11:17:59.179 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:16 ago on Fri 29 Mar 2024 11:15:43 AM UTC. 2024-03-29T11:17:59.260 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-03-29T11:17:59.260 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-03-29T11:17:59.261 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repo Size 2024-03-29T11:17:59.261 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-03-29T11:17:59.261 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-03-29T11:17:59.261 INFO:teuthology.orchestra.run.smithi017.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:17:59.261 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:17:59.261 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-03-29T11:17:59.261 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-03-29T11:17:59.261 INFO:teuthology.orchestra.run.smithi017.stdout:Install 1 Package 2024-03-29T11:17:59.261 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:17:59.261 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 127 k 2024-03-29T11:17:59.261 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 456 k 2024-03-29T11:17:59.261 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-03-29T11:17:59.332 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T11:17:59.333 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T11:17:59.333 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-03-29T11:17:59.333 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T11:17:59.334 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-29T11:17:59.334 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:17:59.334 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:17:59.334 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-29T11:17:59.334 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T11:17:59.334 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-03-29T11:17:59.334 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:17:59.334 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 31 k 2024-03-29T11:17:59.334 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 140 k 2024-03-29T11:17:59.334 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-29T11:17:59.483 INFO:teuthology.orchestra.run.smithi017.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 572 kB/s | 127 kB 00:00 2024-03-29T11:17:59.484 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:59.484 INFO:teuthology.orchestra.run.smithi017.stdout:Total 568 kB/s | 127 kB 00:00 2024-03-29T11:17:59.484 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-03-29T11:17:59.490 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-03-29T11:17:59.490 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-03-29T11:17:59.493 INFO:teuthology.orchestra.run.smithi094.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 196 kB/s | 31 kB 00:00 2024-03-29T11:17:59.494 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:59.494 INFO:teuthology.orchestra.run.smithi094.stdout:Total 194 kB/s | 31 kB 00:00 2024-03-29T11:17:59.494 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-29T11:17:59.497 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-29T11:17:59.498 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-29T11:17:59.520 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-29T11:17:59.521 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-29T11:17:59.548 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-03-29T11:17:59.548 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-03-29T11:17:59.655 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-29T11:17:59.699 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-03-29T11:17:59.760 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:17:59.792 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:00.106 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:00.218 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:00.386 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:00.386 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:18:00.386 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-29T11:18:00.386 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:00.386 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:18:00.386 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T11:18:00.477 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:00.477 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:00.477 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-03-29T11:18:00.478 INFO:teuthology.orchestra.run.smithi017.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:00.478 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:00.478 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-03-29T11:18:00.542 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install librados2 2024-03-29T11:18:00.728 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install libcephfs2 2024-03-29T11:18:01.105 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:18 ago on Fri 29 Mar 2024 11:15:43 AM UTC. 2024-03-29T11:18:01.204 INFO:teuthology.orchestra.run.smithi094.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:01.255 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T11:18:01.256 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-03-29T11:18:01.256 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T11:18:01.305 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install librbd1 2024-03-29T11:18:01.307 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:02:01 ago on Fri 29 Mar 2024 11:16:00 AM UTC. 2024-03-29T11:18:01.409 INFO:teuthology.orchestra.run.smithi017.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:01.467 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-03-29T11:18:01.468 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2024-03-29T11:18:01.468 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-03-29T11:18:01.532 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install libcephfs-devel 2024-03-29T11:18:01.870 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:18 ago on Fri 29 Mar 2024 11:15:43 AM UTC. 2024-03-29T11:18:01.968 INFO:teuthology.orchestra.run.smithi094.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:02.020 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T11:18:02.021 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-03-29T11:18:02.021 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T11:18:02.071 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-rados 2024-03-29T11:18:02.108 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:02:02 ago on Fri 29 Mar 2024 11:16:00 AM UTC. 2024-03-29T11:18:02.265 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-03-29T11:18:02.266 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-03-29T11:18:02.266 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repo Size 2024-03-29T11:18:02.267 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-03-29T11:18:02.267 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-03-29T11:18:02.267 INFO:teuthology.orchestra.run.smithi017.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:18:02.267 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:02.267 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-03-29T11:18:02.267 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-03-29T11:18:02.267 INFO:teuthology.orchestra.run.smithi017.stdout:Install 1 Package 2024-03-29T11:18:02.267 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:02.267 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 31 k 2024-03-29T11:18:02.267 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 140 k 2024-03-29T11:18:02.267 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-03-29T11:18:02.551 INFO:teuthology.orchestra.run.smithi017.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 110 kB/s | 31 kB 00:00 2024-03-29T11:18:02.552 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:02.552 INFO:teuthology.orchestra.run.smithi017.stdout:Total 109 kB/s | 31 kB 00:00 2024-03-29T11:18:02.552 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-03-29T11:18:02.555 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-03-29T11:18:02.555 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-03-29T11:18:02.582 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-03-29T11:18:02.582 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-03-29T11:18:02.633 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:19 ago on Fri 29 Mar 2024 11:15:43 AM UTC. 2024-03-29T11:18:02.709 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-03-29T11:18:02.732 INFO:teuthology.orchestra.run.smithi094.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:02.783 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T11:18:02.784 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-03-29T11:18:02.784 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T11:18:02.829 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-rgw 2024-03-29T11:18:02.830 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:03.212 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:03.400 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:20 ago on Fri 29 Mar 2024 11:15:43 AM UTC. 2024-03-29T11:18:03.472 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:03.472 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:03.472 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-03-29T11:18:03.472 INFO:teuthology.orchestra.run.smithi017.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:03.472 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:03.472 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-03-29T11:18:03.499 INFO:teuthology.orchestra.run.smithi094.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:03.552 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T11:18:03.553 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-03-29T11:18:03.553 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T11:18:03.604 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-cephfs 2024-03-29T11:18:03.626 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install librados2 2024-03-29T11:18:04.173 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:21 ago on Fri 29 Mar 2024 11:15:43 AM UTC. 2024-03-29T11:18:04.196 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:02:04 ago on Fri 29 Mar 2024 11:16:00 AM UTC. 2024-03-29T11:18:04.271 INFO:teuthology.orchestra.run.smithi094.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:04.297 INFO:teuthology.orchestra.run.smithi017.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:04.322 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T11:18:04.323 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-03-29T11:18:04.324 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T11:18:04.352 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-03-29T11:18:04.353 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2024-03-29T11:18:04.353 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-03-29T11:18:04.387 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-rbd 2024-03-29T11:18:04.409 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install librbd1 2024-03-29T11:18:04.949 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:21 ago on Fri 29 Mar 2024 11:15:43 AM UTC. 2024-03-29T11:18:04.983 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:02:05 ago on Fri 29 Mar 2024 11:16:00 AM UTC. 2024-03-29T11:18:05.049 INFO:teuthology.orchestra.run.smithi094.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:05.082 INFO:teuthology.orchestra.run.smithi017.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:05.100 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T11:18:05.101 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-03-29T11:18:05.101 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T11:18:05.136 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-03-29T11:18:05.138 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2024-03-29T11:18:05.138 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-03-29T11:18:05.145 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install rbd-fuse 2024-03-29T11:18:05.189 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install python3-rados 2024-03-29T11:18:05.705 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:22 ago on Fri 29 Mar 2024 11:15:43 AM UTC. 2024-03-29T11:18:05.768 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:02:05 ago on Fri 29 Mar 2024 11:16:00 AM UTC. 2024-03-29T11:18:05.856 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T11:18:05.856 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T11:18:05.857 INFO:teuthology.orchestra.run.smithi094.stdout: Package Architecture Version Repository Size 2024-03-29T11:18:05.857 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T11:18:05.857 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-29T11:18:05.857 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:18:05.857 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:18:05.857 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-29T11:18:05.857 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T11:18:05.857 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-03-29T11:18:05.857 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:18:05.857 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 85 k 2024-03-29T11:18:05.857 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 231 k 2024-03-29T11:18:05.857 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-29T11:18:05.870 INFO:teuthology.orchestra.run.smithi017.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:05.925 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-03-29T11:18:05.926 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2024-03-29T11:18:05.926 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-03-29T11:18:05.976 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install python3-rgw 2024-03-29T11:18:06.146 INFO:teuthology.orchestra.run.smithi094.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 295 kB/s | 85 kB 00:00 2024-03-29T11:18:06.147 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:06.147 INFO:teuthology.orchestra.run.smithi094.stdout:Total 293 kB/s | 85 kB 00:00 2024-03-29T11:18:06.147 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-29T11:18:06.155 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-29T11:18:06.155 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-29T11:18:06.210 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-29T11:18:06.210 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-29T11:18:06.351 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-29T11:18:06.462 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:06.552 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:02:06 ago on Fri 29 Mar 2024 11:16:00 AM UTC. 2024-03-29T11:18:06.659 INFO:teuthology.orchestra.run.smithi017.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:06.715 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-03-29T11:18:06.717 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2024-03-29T11:18:06.717 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-03-29T11:18:06.764 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install python3-cephfs 2024-03-29T11:18:06.862 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:07.113 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:07.113 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:18:07.113 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-29T11:18:07.113 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:07.113 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:18:07.113 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T11:18:07.342 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:02:07 ago on Fri 29 Mar 2024 11:16:00 AM UTC. 2024-03-29T11:18:07.380 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install rbd-mirror 2024-03-29T11:18:07.443 INFO:teuthology.orchestra.run.smithi017.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:07.498 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-03-29T11:18:07.499 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2024-03-29T11:18:07.499 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-03-29T11:18:07.547 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install python3-rbd 2024-03-29T11:18:07.981 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:25 ago on Fri 29 Mar 2024 11:15:43 AM UTC. 2024-03-29T11:18:08.139 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:02:08 ago on Fri 29 Mar 2024 11:16:00 AM UTC. 2024-03-29T11:18:08.140 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T11:18:08.141 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T11:18:08.141 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-03-29T11:18:08.142 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T11:18:08.142 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-29T11:18:08.142 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:18:08.142 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:18:08.142 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-29T11:18:08.142 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T11:18:08.142 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-03-29T11:18:08.142 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:18:08.142 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 3.1 M 2024-03-29T11:18:08.142 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 13 M 2024-03-29T11:18:08.142 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-29T11:18:08.241 INFO:teuthology.orchestra.run.smithi017.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:08.296 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-03-29T11:18:08.297 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2024-03-29T11:18:08.298 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-03-29T11:18:08.352 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install rbd-fuse 2024-03-29T11:18:08.926 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:02:08 ago on Fri 29 Mar 2024 11:16:00 AM UTC. 2024-03-29T11:18:09.083 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-03-29T11:18:09.083 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-03-29T11:18:09.083 INFO:teuthology.orchestra.run.smithi017.stdout: Package Architecture Version Repository Size 2024-03-29T11:18:09.084 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-03-29T11:18:09.084 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-03-29T11:18:09.084 INFO:teuthology.orchestra.run.smithi017.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:18:09.084 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:09.084 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-03-29T11:18:09.084 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-03-29T11:18:09.084 INFO:teuthology.orchestra.run.smithi017.stdout:Install 1 Package 2024-03-29T11:18:09.084 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:09.084 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 85 k 2024-03-29T11:18:09.084 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 231 k 2024-03-29T11:18:09.084 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-03-29T11:18:09.538 INFO:teuthology.orchestra.run.smithi017.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 188 kB/s | 85 kB 00:00 2024-03-29T11:18:09.538 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:09.539 INFO:teuthology.orchestra.run.smithi017.stdout:Total 187 kB/s | 85 kB 00:00 2024-03-29T11:18:09.539 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-03-29T11:18:09.547 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-03-29T11:18:09.547 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-03-29T11:18:09.607 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-03-29T11:18:09.607 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-03-29T11:18:09.757 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-03-29T11:18:09.858 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:10.034 INFO:teuthology.orchestra.run.smithi094.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 1.6 MB/s | 3.1 MB 00:01 2024-03-29T11:18:10.035 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:10.036 INFO:teuthology.orchestra.run.smithi094.stdout:Total 1.6 MB/s | 3.1 MB 00:01 2024-03-29T11:18:10.036 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-29T11:18:10.045 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-29T11:18:10.045 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-29T11:18:10.119 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-29T11:18:10.120 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-29T11:18:10.243 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:10.530 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:10.530 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:10.530 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-03-29T11:18:10.530 INFO:teuthology.orchestra.run.smithi017.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:10.530 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:10.530 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-03-29T11:18:10.558 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-29T11:18:10.627 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:10.651 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:10.651 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:10.651 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:18:10.652 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:18:10.652 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:18:10.652 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:18:10.741 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install rbd-mirror 2024-03-29T11:18:11.316 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:02:11 ago on Fri 29 Mar 2024 11:16:00 AM UTC. 2024-03-29T11:18:11.475 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-03-29T11:18:11.476 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-03-29T11:18:11.476 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repo Size 2024-03-29T11:18:11.476 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-03-29T11:18:11.477 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-03-29T11:18:11.477 INFO:teuthology.orchestra.run.smithi017.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:18:11.477 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:11.477 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-03-29T11:18:11.477 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-03-29T11:18:11.477 INFO:teuthology.orchestra.run.smithi017.stdout:Install 1 Package 2024-03-29T11:18:11.477 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:11.477 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 3.1 M 2024-03-29T11:18:11.477 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 13 M 2024-03-29T11:18:11.477 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-03-29T11:18:11.819 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:11.819 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:18:11.819 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-29T11:18:11.819 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:11.819 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:18:11.820 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T11:18:12.055 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install rbd-nbd 2024-03-29T11:18:12.631 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:29 ago on Fri 29 Mar 2024 11:15:43 AM UTC. 2024-03-29T11:18:12.780 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T11:18:12.781 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T11:18:12.781 INFO:teuthology.orchestra.run.smithi094.stdout: Package Architecture Version Repository Size 2024-03-29T11:18:12.781 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T11:18:12.781 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-29T11:18:12.781 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:18:12.781 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:18:12.781 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-29T11:18:12.781 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T11:18:12.781 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-03-29T11:18:12.781 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:18:12.781 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 171 k 2024-03-29T11:18:12.782 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 494 k 2024-03-29T11:18:12.782 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-29T11:18:13.122 INFO:teuthology.orchestra.run.smithi094.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 505 kB/s | 171 kB 00:00 2024-03-29T11:18:13.122 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:13.123 INFO:teuthology.orchestra.run.smithi094.stdout:Total 502 kB/s | 171 kB 00:00 2024-03-29T11:18:13.123 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-29T11:18:13.131 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-29T11:18:13.131 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-29T11:18:13.186 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-29T11:18:13.187 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-29T11:18:13.340 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-29T11:18:13.454 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:13.659 INFO:teuthology.orchestra.run.smithi017.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 1.4 MB/s | 3.1 MB 00:02 2024-03-29T11:18:13.660 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:13.661 INFO:teuthology.orchestra.run.smithi017.stdout:Total 1.4 MB/s | 3.1 MB 00:02 2024-03-29T11:18:13.661 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-03-29T11:18:13.673 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-03-29T11:18:13.673 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-03-29T11:18:13.753 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-03-29T11:18:13.753 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-03-29T11:18:13.912 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:14.211 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-03-29T11:18:14.217 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:14.217 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:18:14.217 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-29T11:18:14.217 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:14.218 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:18:14.218 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T11:18:14.277 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:14.300 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:14.301 INFO:teuthology.orchestra.run.smithi017.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:14.301 INFO:teuthology.orchestra.run.smithi017.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:18:14.301 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:18:14.301 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:18:14.301 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:14.483 DEBUG:teuthology.parallel:result is None 2024-03-29T11:18:15.455 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:15.455 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:15.455 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-03-29T11:18:15.455 INFO:teuthology.orchestra.run.smithi017.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:15.455 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:15.456 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-03-29T11:18:15.713 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install rbd-nbd 2024-03-29T11:18:16.291 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:02:16 ago on Fri 29 Mar 2024 11:16:00 AM UTC. 2024-03-29T11:18:16.446 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-03-29T11:18:16.449 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-03-29T11:18:16.449 INFO:teuthology.orchestra.run.smithi017.stdout: Package Architecture Version Repository Size 2024-03-29T11:18:16.449 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-03-29T11:18:16.449 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-03-29T11:18:16.450 INFO:teuthology.orchestra.run.smithi017.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:18:16.450 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:16.450 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-03-29T11:18:16.450 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-03-29T11:18:16.450 INFO:teuthology.orchestra.run.smithi017.stdout:Install 1 Package 2024-03-29T11:18:16.450 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:16.450 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 171 k 2024-03-29T11:18:16.450 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 494 k 2024-03-29T11:18:16.450 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-03-29T11:18:17.571 INFO:teuthology.orchestra.run.smithi017.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 153 kB/s | 171 kB 00:01 2024-03-29T11:18:17.571 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:17.572 INFO:teuthology.orchestra.run.smithi017.stdout:Total 152 kB/s | 171 kB 00:01 2024-03-29T11:18:17.572 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-03-29T11:18:17.581 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-03-29T11:18:17.581 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-03-29T11:18:17.641 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-03-29T11:18:17.642 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-03-29T11:18:17.798 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-03-29T11:18:17.908 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:18.334 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:18.620 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:18.620 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:18.620 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-03-29T11:18:18.620 INFO:teuthology.orchestra.run.smithi017.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:18.620 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:18.621 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-03-29T11:18:18.841 DEBUG:teuthology.parallel:result is None 2024-03-29T11:18:18.842 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:18:19.063 DEBUG:teuthology.orchestra.run.smithi017:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:18:19.084 INFO:teuthology.orchestra.run.smithi017.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:18:19.085 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:18:19.085 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:18:19.087 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:18:19.227 DEBUG:teuthology.orchestra.run.smithi094:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:18:19.251 INFO:teuthology.orchestra.run.smithi094.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:18:19.251 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:18:19.252 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:18:19.253 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T11:18:19.254 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-03-29T11:18:19.254 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:18:19.283 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-29T11:18:19.283 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:18:19.322 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T11:18:19.322 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-03-29T11:18:19.322 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:18:19.350 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:18:19.416 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-29T11:18:19.416 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:18:19.445 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:18:19.514 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T11:18:19.514 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-03-29T11:18:19.515 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:18:19.541 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:18:19.608 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-29T11:18:19.608 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:18:19.636 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:18:19.704 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T11:18:19.705 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-03-29T11:18:19.705 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:18:19.731 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:18:19.798 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-29T11:18:19.798 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:18:19.825 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:18:19.892 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T11:18:19.978 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T11:18:19.978 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 777 /var/log/ceph 2024-03-29T11:18:19.980 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 777 /var/log/ceph 2024-03-29T11:18:20.007 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T11:18:20.007 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:18:20.047 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:18:20.074 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T11:18:20.075 DEBUG:teuthology.orchestra.run.smithi017:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:18:20.113 DEBUG:teuthology.orchestra.run.smithi094:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:18:20.151 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T11:18:20.151 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T11:18:20.151 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622683', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622683', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi017.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGZcCOJjiFUA2a6jFFSoq93F3qAol6z1qPchs7Cw7fNOZHqYW7tnklnfaMxeM5zo+PIdJdeOTOw9XjhdIP5sXs0=', 'smithi094.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFplVfDj+ocYFtAOZXcnsLnocc/g2p4HL0RJGa2ZhOlaiDHJzgpAQBJR8Rqbd8sQotCrDibgV/AXAR1qM2gc4VI='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T11:18:20.151 DEBUG:teuthology.orchestra.run.smithi017:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:18:20.180 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:18:20.196 DEBUG:teuthology.orchestra.run.smithi017:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:18:20.234 DEBUG:teuthology.orchestra.run.smithi094:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:18:20.263 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-03-29T11:18:20.264 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:18:20.312 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:18:20.313 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_1 2024-03-29T11:18:20.367 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:18:20.367 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:20.367 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 5h/5d Inode: 661 Links: 1 2024-03-29T11:18:20.368 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:20.368 INFO:teuthology.orchestra.run.smithi017.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:20.368 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2024-03-29 11:18:14.564636390 +0000 2024-03-29T11:18:20.368 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2024-03-29 11:14:47.101134702 +0000 2024-03-29T11:18:20.368 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2024-03-29 11:14:47.101134702 +0000 2024-03-29T11:18:20.368 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: 2024-03-29 11:14:47.101134702 +0000 2024-03-29T11:18:20.368 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:18:20.432 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2024-03-29T11:18:20.433 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2024-03-29T11:18:20.433 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 7.7451e-05 s, 6.6 MB/s 2024-03-29T11:18:20.434 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:18:20.489 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_2 2024-03-29T11:18:20.543 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:18:20.543 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:20.543 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 5h/5d Inode: 680 Links: 1 2024-03-29T11:18:20.543 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:20.543 INFO:teuthology.orchestra.run.smithi017.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:20.543 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2024-03-29 11:18:14.564636390 +0000 2024-03-29T11:18:20.543 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2024-03-29 11:14:47.434131343 +0000 2024-03-29T11:18:20.543 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2024-03-29 11:14:47.434131343 +0000 2024-03-29T11:18:20.544 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: 2024-03-29 11:14:47.434131343 +0000 2024-03-29T11:18:20.546 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:18:20.607 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2024-03-29T11:18:20.607 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2024-03-29T11:18:20.607 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 9.5101e-05 s, 5.4 MB/s 2024-03-29T11:18:20.608 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:18:20.664 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_3 2024-03-29T11:18:20.719 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:18:20.719 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:20.719 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-03-29T11:18:20.719 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:20.719 INFO:teuthology.orchestra.run.smithi017.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:20.719 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2024-03-29 11:18:14.564636390 +0000 2024-03-29T11:18:20.719 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2024-03-29 11:14:47.778127873 +0000 2024-03-29T11:18:20.719 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2024-03-29 11:14:47.778127873 +0000 2024-03-29T11:18:20.719 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: 2024-03-29 11:14:47.778127873 +0000 2024-03-29T11:18:20.720 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:18:20.783 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2024-03-29T11:18:20.784 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2024-03-29T11:18:20.784 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 9.464e-05 s, 5.4 MB/s 2024-03-29T11:18:20.785 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:18:20.840 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_4 2024-03-29T11:18:20.894 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:18:20.894 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:20.894 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-03-29T11:18:20.894 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:20.894 INFO:teuthology.orchestra.run.smithi017.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:20.894 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2024-03-29 11:18:14.565636379 +0000 2024-03-29T11:18:20.895 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2024-03-29 11:14:48.134124282 +0000 2024-03-29T11:18:20.895 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2024-03-29 11:14:48.134124282 +0000 2024-03-29T11:18:20.895 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: 2024-03-29 11:14:48.134124282 +0000 2024-03-29T11:18:20.895 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:18:20.960 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2024-03-29T11:18:20.960 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2024-03-29T11:18:20.960 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 9.4394e-05 s, 5.4 MB/s 2024-03-29T11:18:20.961 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:18:21.017 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:18:21.017 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-29T11:18:21.017 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:18:21.034 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:18:21.035 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_1 2024-03-29T11:18:21.091 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:18:21.091 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:21.091 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-29T11:18:21.091 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:21.091 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:21.091 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-03-29 11:18:10.905365853 +0000 2024-03-29T11:18:21.091 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-03-29 11:14:46.604211518 +0000 2024-03-29T11:18:21.091 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-03-29 11:14:46.604211518 +0000 2024-03-29T11:18:21.091 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: 2024-03-29 11:14:46.604211518 +0000 2024-03-29T11:18:21.092 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:18:21.158 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-03-29T11:18:21.158 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-03-29T11:18:21.158 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.000109907 s, 4.7 MB/s 2024-03-29T11:18:21.159 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:18:21.217 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_2 2024-03-29T11:18:21.272 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:18:21.272 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:21.272 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-29T11:18:21.273 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:21.273 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:21.273 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-03-29 11:18:10.905365853 +0000 2024-03-29T11:18:21.273 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-03-29 11:14:47.132202506 +0000 2024-03-29T11:18:21.273 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-03-29 11:14:47.132202506 +0000 2024-03-29T11:18:21.273 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: 2024-03-29 11:14:47.132202506 +0000 2024-03-29T11:18:21.273 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:18:21.338 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-03-29T11:18:21.338 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-03-29T11:18:21.339 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 8.0876e-05 s, 6.3 MB/s 2024-03-29T11:18:21.340 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:18:21.396 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_3 2024-03-29T11:18:21.452 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:18:21.452 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:21.452 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-29T11:18:21.452 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:21.452 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:21.452 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-03-29 11:18:10.906365836 +0000 2024-03-29T11:18:21.452 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-03-29 11:14:47.480196565 +0000 2024-03-29T11:18:21.453 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-03-29 11:14:47.480196565 +0000 2024-03-29T11:18:21.453 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: 2024-03-29 11:14:47.480196565 +0000 2024-03-29T11:18:21.453 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:18:21.518 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-03-29T11:18:21.518 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-03-29T11:18:21.518 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.000100538 s, 5.1 MB/s 2024-03-29T11:18:21.519 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:18:21.575 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_4 2024-03-29T11:18:21.630 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:18:21.630 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:21.630 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-29T11:18:21.630 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:21.631 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:21.631 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-03-29 11:18:10.906365836 +0000 2024-03-29T11:18:21.631 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-03-29 11:14:47.813190881 +0000 2024-03-29T11:18:21.631 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-03-29 11:14:47.813190881 +0000 2024-03-29T11:18:21.631 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: 2024-03-29 11:14:47.813190881 +0000 2024-03-29T11:18:21.631 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:18:21.696 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-03-29T11:18:21.696 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-03-29T11:18:21.696 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.00010341 s, 5.0 MB/s 2024-03-29T11:18:21.697 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:18:21.755 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:18:21.755 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi017.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi094.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:18:21.755 INFO:tasks.ceph:Generating config... 2024-03-29T11:18:21.756 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T11:18:21.756 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T11:18:21.756 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T11:18:21.756 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T11:18:21.757 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T11:18:21.757 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T11:18:21.757 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T11:18:21.757 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T11:18:21.757 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T11:18:21.757 INFO:tasks.ceph:[global] ms type = async 2024-03-29T11:18:21.757 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T11:18:21.757 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T11:18:21.757 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T11:18:21.757 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T11:18:21.757 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T11:18:21.757 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T11:18:21.758 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T11:18:21.758 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T11:18:21.758 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T11:18:21.758 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T11:18:21.758 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T11:18:21.758 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T11:18:21.758 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T11:18:21.758 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-03-29T11:18:21.758 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T11:18:21.758 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T11:18:21.758 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T11:18:21.758 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T11:18:21.758 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T11:18:21.759 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T11:18:21.759 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T11:18:21.759 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T11:18:21.759 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T11:18:21.759 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T11:18:21.759 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T11:18:21.759 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T11:18:21.759 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T11:18:21.759 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T11:18:21.759 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T11:18:21.760 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T11:18:21.760 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T11:18:21.760 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T11:18:21.760 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T11:18:21.760 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T11:18:21.760 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T11:18:21.760 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T11:18:21.760 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T11:18:21.760 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T11:18:21.761 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T11:18:21.761 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T11:18:21.816 INFO:teuthology.orchestra.run.smithi017.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T11:18:21.819 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T11:18:21.866 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:18:21.894 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.17:6789]'), ('mon.c', '[v1:172.21.15.17:6790]'), ('mon.b', '[v1:172.21.15.94:6789]')] 2024-03-29T11:18:21.895 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.17:6789],[v1:172.21.15.17:6790],[v1:172.21.15.94:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T11:18:21.895 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-03-29T11:18:21.896 DEBUG:teuthology.orchestra.run.smithi017:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:18:21.950 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.17:6789]' --addv c '[v1:172.21.15.17:6790]' --addv b '[v1:172.21.15.94:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:22.036 INFO:teuthology.orchestra.run.smithi017.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:22.037 INFO:teuthology.orchestra.run.smithi017.stdout:monmaptool: generated fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:22.037 INFO:teuthology.orchestra.run.smithi017.stdout:setting min_mon_release = quincy 2024-03-29T11:18:22.037 INFO:teuthology.orchestra.run.smithi017.stdout:epoch 0 2024-03-29T11:18:22.037 INFO:teuthology.orchestra.run.smithi017.stdout:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:22.037 INFO:teuthology.orchestra.run.smithi017.stdout:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:22.037 INFO:teuthology.orchestra.run.smithi017.stdout:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:22.037 INFO:teuthology.orchestra.run.smithi017.stdout:min_mon_release 17 (quincy) 2024-03-29T11:18:22.037 INFO:teuthology.orchestra.run.smithi017.stdout:election_strategy: 1 2024-03-29T11:18:22.037 INFO:teuthology.orchestra.run.smithi017.stdout:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:22.037 INFO:teuthology.orchestra.run.smithi017.stdout:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:22.037 INFO:teuthology.orchestra.run.smithi017.stdout:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:22.038 INFO:teuthology.orchestra.run.smithi017.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T11:18:22.039 DEBUG:teuthology.orchestra.run.smithi017:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:18:22.053 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID a932e4f9-b3f4-4d39-b76e-25f5220acc19... 2024-03-29T11:18:22.054 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:18:22.096 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:18:22.144 INFO:teuthology.orchestra.run.smithi017.stdout:[global] 2024-03-29T11:18:22.144 INFO:teuthology.orchestra.run.smithi017.stdout: chdir = "" 2024-03-29T11:18:22.144 INFO:teuthology.orchestra.run.smithi017.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:18:22.144 INFO:teuthology.orchestra.run.smithi017.stdout: auth supported = cephx 2024-03-29T11:18:22.144 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:22.145 INFO:teuthology.orchestra.run.smithi017.stdout: filestore xattr use omap = true 2024-03-29T11:18:22.145 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:22.145 INFO:teuthology.orchestra.run.smithi017.stdout: mon clock drift allowed = 1.000 2024-03-29T11:18:22.145 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:22.145 INFO:teuthology.orchestra.run.smithi017.stdout: osd crush chooseleaf type = 0 2024-03-29T11:18:22.145 INFO:teuthology.orchestra.run.smithi017.stdout: auth debug = true 2024-03-29T11:18:22.145 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:22.145 INFO:teuthology.orchestra.run.smithi017.stdout: ms die on old message = true 2024-03-29T11:18:22.145 INFO:teuthology.orchestra.run.smithi017.stdout: ms die on bug = true 2024-03-29T11:18:22.145 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:22.145 INFO:teuthology.orchestra.run.smithi017.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:18:22.145 INFO:teuthology.orchestra.run.smithi017.stdout: mon pg warn max object skew = 0 2024-03-29T11:18:22.146 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:22.146 INFO:teuthology.orchestra.run.smithi017.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:18:22.146 INFO:teuthology.orchestra.run.smithi017.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:18:22.146 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:22.146 INFO:teuthology.orchestra.run.smithi017.stdout: osd pool default size = 2 2024-03-29T11:18:22.146 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:22.146 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd allow primary affinity = true 2024-03-29T11:18:22.146 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd allow pg remap = true 2024-03-29T11:18:22.146 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:18:22.146 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:18:22.146 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on no sortbitwise = false 2024-03-29T11:18:22.146 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:18:22.147 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on too few osds = false 2024-03-29T11:18:22.147 INFO:teuthology.orchestra.run.smithi017.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:18:22.147 INFO:teuthology.orchestra.run.smithi017.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:18:22.147 INFO:teuthology.orchestra.run.smithi017.stdout: mon_allow_pool_size_one = true 2024-03-29T11:18:22.147 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:22.147 INFO:teuthology.orchestra.run.smithi017.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:18:22.147 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:22.147 INFO:teuthology.orchestra.run.smithi017.stdout: osd default data pool replay window = 5 2024-03-29T11:18:22.148 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:22.148 INFO:teuthology.orchestra.run.smithi017.stdout: mon allow pool delete = true 2024-03-29T11:18:22.148 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:22.148 INFO:teuthology.orchestra.run.smithi017.stdout: mon cluster log file level = debug 2024-03-29T11:18:22.148 INFO:teuthology.orchestra.run.smithi017.stdout: debug asserts on shutdown = true 2024-03-29T11:18:22.148 INFO:teuthology.orchestra.run.smithi017.stdout: mon health detail to clog = false 2024-03-29T11:18:22.148 INFO:teuthology.orchestra.run.smithi017.stdout: mon host = "[v1:172.21.15.17:6789],[v1:172.21.15.17:6790],[v1:172.21.15.94:6789]" 2024-03-29T11:18:22.148 INFO:teuthology.orchestra.run.smithi017.stdout: lockdep = True 2024-03-29T11:18:22.149 INFO:teuthology.orchestra.run.smithi017.stdout: mon client directed command retry = 5 2024-03-29T11:18:22.149 INFO:teuthology.orchestra.run.smithi017.stdout: mon election default strategy = 1 2024-03-29T11:18:22.149 INFO:teuthology.orchestra.run.smithi017.stdout: ms bind msgr2 = False 2024-03-29T11:18:22.149 INFO:teuthology.orchestra.run.smithi017.stdout: ms inject socket failures = 5000 2024-03-29T11:18:22.149 INFO:teuthology.orchestra.run.smithi017.stdout: ms type = async 2024-03-29T11:18:22.149 INFO:teuthology.orchestra.run.smithi017.stdout: fsid = a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:22.149 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:22.149 INFO:teuthology.orchestra.run.smithi017.stdout:[osd] 2024-03-29T11:18:22.149 INFO:teuthology.orchestra.run.smithi017.stdout: osd journal size = 100 2024-03-29T11:18:22.149 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:22.149 INFO:teuthology.orchestra.run.smithi017.stdout: osd scrub load threshold = 5.0 2024-03-29T11:18:22.149 INFO:teuthology.orchestra.run.smithi017.stdout: osd scrub max interval = 600 2024-03-29T11:18:22.150 INFO:teuthology.orchestra.run.smithi017.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:18:22.150 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:22.150 INFO:teuthology.orchestra.run.smithi017.stdout: osd recover clone overlap = true 2024-03-29T11:18:22.150 INFO:teuthology.orchestra.run.smithi017.stdout: osd recovery max chunk = 1048576 2024-03-29T11:18:22.150 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:22.150 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug shutdown = true 2024-03-29T11:18:22.150 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug op order = true 2024-03-29T11:18:22.150 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug verify stray on activate = true 2024-03-29T11:18:22.150 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug trim objects = true 2024-03-29T11:18:22.150 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:22.150 INFO:teuthology.orchestra.run.smithi017.stdout: osd open classes on start = true 2024-03-29T11:18:22.151 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug pg log writeout = true 2024-03-29T11:18:22.151 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:22.151 INFO:teuthology.orchestra.run.smithi017.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:18:22.151 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:22.151 INFO:teuthology.orchestra.run.smithi017.stdout: osd map max advance = 10 2024-03-29T11:18:22.151 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:22.151 INFO:teuthology.orchestra.run.smithi017.stdout: journal zero on create = true 2024-03-29T11:18:22.151 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:22.151 INFO:teuthology.orchestra.run.smithi017.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:18:22.151 INFO:teuthology.orchestra.run.smithi017.stdout: filestore apply finisher threads = 3 2024-03-29T11:18:22.151 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:22.151 INFO:teuthology.orchestra.run.smithi017.stdout: bdev debug aio = true 2024-03-29T11:18:22.151 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug misdirected ops = true 2024-03-29T11:18:22.152 INFO:teuthology.orchestra.run.smithi017.stdout: bluestore block size = 96636764160 2024-03-29T11:18:22.152 INFO:teuthology.orchestra.run.smithi017.stdout: bluestore compression algorithm = zstd 2024-03-29T11:18:22.152 INFO:teuthology.orchestra.run.smithi017.stdout: bluestore compression mode = aggressive 2024-03-29T11:18:22.152 INFO:teuthology.orchestra.run.smithi017.stdout: bluestore fsck on mount = True 2024-03-29T11:18:22.152 INFO:teuthology.orchestra.run.smithi017.stdout: bluestore zero block detection = True 2024-03-29T11:18:22.152 INFO:teuthology.orchestra.run.smithi017.stdout: debug bluefs = 20 2024-03-29T11:18:22.152 INFO:teuthology.orchestra.run.smithi017.stdout: debug bluestore = 20 2024-03-29T11:18:22.152 INFO:teuthology.orchestra.run.smithi017.stdout: debug monc = 20 2024-03-29T11:18:22.152 INFO:teuthology.orchestra.run.smithi017.stdout: debug ms = 1 2024-03-29T11:18:22.152 INFO:teuthology.orchestra.run.smithi017.stdout: debug osd = 20 2024-03-29T11:18:22.152 INFO:teuthology.orchestra.run.smithi017.stdout: debug rocksdb = 10 2024-03-29T11:18:22.152 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:18:22.153 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd full ratio = 0.9 2024-03-29T11:18:22.153 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:18:22.153 INFO:teuthology.orchestra.run.smithi017.stdout: osd class default list = * 2024-03-29T11:18:22.153 INFO:teuthology.orchestra.run.smithi017.stdout: osd class load list = * 2024-03-29T11:18:22.153 INFO:teuthology.orchestra.run.smithi017.stdout: osd client watch timeout = 120 2024-03-29T11:18:22.153 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug verify cached snaps = True 2024-03-29T11:18:22.153 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug verify missing on start = True 2024-03-29T11:18:22.153 INFO:teuthology.orchestra.run.smithi017.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:18:22.153 INFO:teuthology.orchestra.run.smithi017.stdout: osd mclock override recovery settings = True 2024-03-29T11:18:22.153 INFO:teuthology.orchestra.run.smithi017.stdout: osd objectstore = bluestore 2024-03-29T11:18:22.154 INFO:teuthology.orchestra.run.smithi017.stdout: osd op queue = debug_random 2024-03-29T11:18:22.154 INFO:teuthology.orchestra.run.smithi017.stdout: osd op queue cut off = debug_random 2024-03-29T11:18:22.154 INFO:teuthology.orchestra.run.smithi017.stdout: osd shutdown pgref assert = True 2024-03-29T11:18:22.154 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:22.154 INFO:teuthology.orchestra.run.smithi017.stdout:[mgr] 2024-03-29T11:18:22.154 INFO:teuthology.orchestra.run.smithi017.stdout: debug ms = 1 2024-03-29T11:18:22.154 INFO:teuthology.orchestra.run.smithi017.stdout: debug mgr = 20 2024-03-29T11:18:22.154 INFO:teuthology.orchestra.run.smithi017.stdout: debug mon = 20 2024-03-29T11:18:22.155 INFO:teuthology.orchestra.run.smithi017.stdout: debug auth = 20 2024-03-29T11:18:22.155 INFO:teuthology.orchestra.run.smithi017.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:18:22.155 INFO:teuthology.orchestra.run.smithi017.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:18:22.155 INFO:teuthology.orchestra.run.smithi017.stdout: mgr/telemetry/nag = false 2024-03-29T11:18:22.155 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:22.155 INFO:teuthology.orchestra.run.smithi017.stdout:[mon] 2024-03-29T11:18:22.155 INFO:teuthology.orchestra.run.smithi017.stdout: debug ms = 1 2024-03-29T11:18:22.155 INFO:teuthology.orchestra.run.smithi017.stdout: debug mon = 20 2024-03-29T11:18:22.155 INFO:teuthology.orchestra.run.smithi017.stdout: debug paxos = 20 2024-03-29T11:18:22.155 INFO:teuthology.orchestra.run.smithi017.stdout: debug auth = 20 2024-03-29T11:18:22.155 INFO:teuthology.orchestra.run.smithi017.stdout: mon data avail warn = 5 2024-03-29T11:18:22.155 INFO:teuthology.orchestra.run.smithi017.stdout: mon mgr mkfs grace = 240 2024-03-29T11:18:22.156 INFO:teuthology.orchestra.run.smithi017.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:18:22.156 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd reporter subtree level = osd 2024-03-29T11:18:22.156 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd prime pg temp = true 2024-03-29T11:18:22.156 INFO:teuthology.orchestra.run.smithi017.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:18:22.156 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:22.156 INFO:teuthology.orchestra.run.smithi017.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:18:22.156 INFO:teuthology.orchestra.run.smithi017.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:18:22.156 INFO:teuthology.orchestra.run.smithi017.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:18:22.156 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:22.156 INFO:teuthology.orchestra.run.smithi017.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:18:22.156 INFO:teuthology.orchestra.run.smithi017.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:18:22.156 INFO:teuthology.orchestra.run.smithi017.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:18:22.157 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:22.157 INFO:teuthology.orchestra.run.smithi017.stdout: # 1m isn't quite enough 2024-03-29T11:18:22.157 INFO:teuthology.orchestra.run.smithi017.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:18:22.157 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:22.157 INFO:teuthology.orchestra.run.smithi017.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:18:22.157 INFO:teuthology.orchestra.run.smithi017.stdout: mon min osdmap epochs = 50 2024-03-29T11:18:22.157 INFO:teuthology.orchestra.run.smithi017.stdout: mon osdmap full prune interval = 2 2024-03-29T11:18:22.157 INFO:teuthology.orchestra.run.smithi017.stdout: mon osdmap full prune min = 15 2024-03-29T11:18:22.157 INFO:teuthology.orchestra.run.smithi017.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:18:22.157 INFO:teuthology.orchestra.run.smithi017.stdout: mon scrub interval = 300 2024-03-29T11:18:22.157 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on pool no app = False 2024-03-29T11:18:22.157 INFO:teuthology.orchestra.run.smithi017.stdout: paxos service trim min = 10 2024-03-29T11:18:22.158 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:22.158 INFO:teuthology.orchestra.run.smithi017.stdout:[client] 2024-03-29T11:18:22.158 INFO:teuthology.orchestra.run.smithi017.stdout: rgw cache enabled = true 2024-03-29T11:18:22.158 INFO:teuthology.orchestra.run.smithi017.stdout: rgw enable ops log = true 2024-03-29T11:18:22.158 INFO:teuthology.orchestra.run.smithi017.stdout: rgw enable usage log = true 2024-03-29T11:18:22.158 INFO:teuthology.orchestra.run.smithi017.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:18:22.158 INFO:teuthology.orchestra.run.smithi017.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:18:22.158 INFO:teuthology.orchestra.run.smithi017.stdout: debug monc = 20 2024-03-29T11:18:22.159 INFO:teuthology.orchestra.run.smithi017.stdout: debug ms = 1 2024-03-29T11:18:22.159 INFO:teuthology.orchestra.run.smithi017.stdout: debug objecter = 20 2024-03-29T11:18:22.159 INFO:teuthology.orchestra.run.smithi017.stdout: debug rados = 20 2024-03-29T11:18:22.159 INFO:teuthology.orchestra.run.smithi017.stdout:[mon.a] 2024-03-29T11:18:22.159 INFO:teuthology.orchestra.run.smithi017.stdout:[mon.c] 2024-03-29T11:18:22.159 INFO:teuthology.orchestra.run.smithi017.stdout:[mon.b] 2024-03-29T11:18:22.161 INFO:teuthology.orchestra.run.smithi094.stdout:[global] 2024-03-29T11:18:22.161 INFO:teuthology.orchestra.run.smithi094.stdout: chdir = "" 2024-03-29T11:18:22.161 INFO:teuthology.orchestra.run.smithi094.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:18:22.161 INFO:teuthology.orchestra.run.smithi094.stdout: auth supported = cephx 2024-03-29T11:18:22.161 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:18:22.161 INFO:teuthology.orchestra.run.smithi094.stdout: filestore xattr use omap = true 2024-03-29T11:18:22.162 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:18:22.162 INFO:teuthology.orchestra.run.smithi094.stdout: mon clock drift allowed = 1.000 2024-03-29T11:18:22.162 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:18:22.162 INFO:teuthology.orchestra.run.smithi094.stdout: osd crush chooseleaf type = 0 2024-03-29T11:18:22.162 INFO:teuthology.orchestra.run.smithi094.stdout: auth debug = true 2024-03-29T11:18:22.162 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:18:22.162 INFO:teuthology.orchestra.run.smithi094.stdout: ms die on old message = true 2024-03-29T11:18:22.162 INFO:teuthology.orchestra.run.smithi094.stdout: ms die on bug = true 2024-03-29T11:18:22.162 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:18:22.162 INFO:teuthology.orchestra.run.smithi094.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:18:22.162 INFO:teuthology.orchestra.run.smithi094.stdout: mon pg warn max object skew = 0 2024-03-29T11:18:22.162 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:18:22.163 INFO:teuthology.orchestra.run.smithi094.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:18:22.163 INFO:teuthology.orchestra.run.smithi094.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:18:22.163 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:18:22.163 INFO:teuthology.orchestra.run.smithi094.stdout: osd pool default size = 2 2024-03-29T11:18:22.163 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:18:22.163 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd allow primary affinity = true 2024-03-29T11:18:22.163 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd allow pg remap = true 2024-03-29T11:18:22.163 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:18:22.163 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:18:22.163 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on no sortbitwise = false 2024-03-29T11:18:22.163 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:18:22.163 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on too few osds = false 2024-03-29T11:18:22.164 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:18:22.164 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:18:22.164 INFO:teuthology.orchestra.run.smithi094.stdout: mon_allow_pool_size_one = true 2024-03-29T11:18:22.164 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:18:22.164 INFO:teuthology.orchestra.run.smithi094.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:18:22.164 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:18:22.164 INFO:teuthology.orchestra.run.smithi094.stdout: osd default data pool replay window = 5 2024-03-29T11:18:22.164 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:18:22.164 INFO:teuthology.orchestra.run.smithi094.stdout: mon allow pool delete = true 2024-03-29T11:18:22.164 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:18:22.164 INFO:teuthology.orchestra.run.smithi094.stdout: mon cluster log file level = debug 2024-03-29T11:18:22.165 INFO:teuthology.orchestra.run.smithi094.stdout: debug asserts on shutdown = true 2024-03-29T11:18:22.165 INFO:teuthology.orchestra.run.smithi094.stdout: mon health detail to clog = false 2024-03-29T11:18:22.165 INFO:teuthology.orchestra.run.smithi094.stdout: mon host = "[v1:172.21.15.17:6789],[v1:172.21.15.17:6790],[v1:172.21.15.94:6789]" 2024-03-29T11:18:22.165 INFO:teuthology.orchestra.run.smithi094.stdout: lockdep = True 2024-03-29T11:18:22.165 INFO:teuthology.orchestra.run.smithi094.stdout: mon client directed command retry = 5 2024-03-29T11:18:22.165 INFO:teuthology.orchestra.run.smithi094.stdout: mon election default strategy = 1 2024-03-29T11:18:22.165 INFO:teuthology.orchestra.run.smithi094.stdout: ms bind msgr2 = False 2024-03-29T11:18:22.165 INFO:teuthology.orchestra.run.smithi094.stdout: ms inject socket failures = 5000 2024-03-29T11:18:22.165 INFO:teuthology.orchestra.run.smithi094.stdout: ms type = async 2024-03-29T11:18:22.165 INFO:teuthology.orchestra.run.smithi094.stdout: fsid = a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:22.165 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:18:22.165 INFO:teuthology.orchestra.run.smithi094.stdout:[osd] 2024-03-29T11:18:22.166 INFO:teuthology.orchestra.run.smithi094.stdout: osd journal size = 100 2024-03-29T11:18:22.166 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:18:22.166 INFO:teuthology.orchestra.run.smithi094.stdout: osd scrub load threshold = 5.0 2024-03-29T11:18:22.166 INFO:teuthology.orchestra.run.smithi094.stdout: osd scrub max interval = 600 2024-03-29T11:18:22.166 INFO:teuthology.orchestra.run.smithi094.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:18:22.166 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:18:22.166 INFO:teuthology.orchestra.run.smithi094.stdout: osd recover clone overlap = true 2024-03-29T11:18:22.166 INFO:teuthology.orchestra.run.smithi094.stdout: osd recovery max chunk = 1048576 2024-03-29T11:18:22.166 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:18:22.166 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug shutdown = true 2024-03-29T11:18:22.166 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug op order = true 2024-03-29T11:18:22.166 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug verify stray on activate = true 2024-03-29T11:18:22.167 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug trim objects = true 2024-03-29T11:18:22.167 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:18:22.167 INFO:teuthology.orchestra.run.smithi094.stdout: osd open classes on start = true 2024-03-29T11:18:22.167 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug pg log writeout = true 2024-03-29T11:18:22.167 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:18:22.167 INFO:teuthology.orchestra.run.smithi094.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:18:22.167 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:18:22.167 INFO:teuthology.orchestra.run.smithi094.stdout: osd map max advance = 10 2024-03-29T11:18:22.167 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:18:22.167 INFO:teuthology.orchestra.run.smithi094.stdout: journal zero on create = true 2024-03-29T11:18:22.167 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:18:22.167 INFO:teuthology.orchestra.run.smithi094.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:18:22.167 INFO:teuthology.orchestra.run.smithi094.stdout: filestore apply finisher threads = 3 2024-03-29T11:18:22.168 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:18:22.168 INFO:teuthology.orchestra.run.smithi094.stdout: bdev debug aio = true 2024-03-29T11:18:22.168 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug misdirected ops = true 2024-03-29T11:18:22.168 INFO:teuthology.orchestra.run.smithi094.stdout: bluestore block size = 96636764160 2024-03-29T11:18:22.168 INFO:teuthology.orchestra.run.smithi094.stdout: bluestore compression algorithm = zstd 2024-03-29T11:18:22.168 INFO:teuthology.orchestra.run.smithi094.stdout: bluestore compression mode = aggressive 2024-03-29T11:18:22.168 INFO:teuthology.orchestra.run.smithi094.stdout: bluestore fsck on mount = True 2024-03-29T11:18:22.168 INFO:teuthology.orchestra.run.smithi094.stdout: bluestore zero block detection = True 2024-03-29T11:18:22.168 INFO:teuthology.orchestra.run.smithi094.stdout: debug bluefs = 20 2024-03-29T11:18:22.168 INFO:teuthology.orchestra.run.smithi094.stdout: debug bluestore = 20 2024-03-29T11:18:22.168 INFO:teuthology.orchestra.run.smithi094.stdout: debug monc = 20 2024-03-29T11:18:22.169 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 1 2024-03-29T11:18:22.169 INFO:teuthology.orchestra.run.smithi094.stdout: debug osd = 20 2024-03-29T11:18:22.169 INFO:teuthology.orchestra.run.smithi094.stdout: debug rocksdb = 10 2024-03-29T11:18:22.169 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:18:22.169 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd full ratio = 0.9 2024-03-29T11:18:22.170 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:18:22.170 INFO:teuthology.orchestra.run.smithi094.stdout: osd class default list = * 2024-03-29T11:18:22.170 INFO:teuthology.orchestra.run.smithi094.stdout: osd class load list = * 2024-03-29T11:18:22.170 INFO:teuthology.orchestra.run.smithi094.stdout: osd client watch timeout = 120 2024-03-29T11:18:22.170 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug verify cached snaps = True 2024-03-29T11:18:22.170 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug verify missing on start = True 2024-03-29T11:18:22.170 INFO:teuthology.orchestra.run.smithi094.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:18:22.171 INFO:teuthology.orchestra.run.smithi094.stdout: osd mclock override recovery settings = True 2024-03-29T11:18:22.171 INFO:teuthology.orchestra.run.smithi094.stdout: osd objectstore = bluestore 2024-03-29T11:18:22.171 INFO:teuthology.orchestra.run.smithi094.stdout: osd op queue = debug_random 2024-03-29T11:18:22.171 INFO:teuthology.orchestra.run.smithi094.stdout: osd op queue cut off = debug_random 2024-03-29T11:18:22.171 INFO:teuthology.orchestra.run.smithi094.stdout: osd shutdown pgref assert = True 2024-03-29T11:18:22.171 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:18:22.172 INFO:teuthology.orchestra.run.smithi094.stdout:[mgr] 2024-03-29T11:18:22.172 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 1 2024-03-29T11:18:22.172 INFO:teuthology.orchestra.run.smithi094.stdout: debug mgr = 20 2024-03-29T11:18:22.172 INFO:teuthology.orchestra.run.smithi094.stdout: debug mon = 20 2024-03-29T11:18:22.172 INFO:teuthology.orchestra.run.smithi094.stdout: debug auth = 20 2024-03-29T11:18:22.172 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:18:22.172 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:18:22.172 INFO:teuthology.orchestra.run.smithi094.stdout: mgr/telemetry/nag = false 2024-03-29T11:18:22.173 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:18:22.173 INFO:teuthology.orchestra.run.smithi094.stdout:[mon] 2024-03-29T11:18:22.173 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 1 2024-03-29T11:18:22.173 INFO:teuthology.orchestra.run.smithi094.stdout: debug mon = 20 2024-03-29T11:18:22.173 INFO:teuthology.orchestra.run.smithi094.stdout: debug paxos = 20 2024-03-29T11:18:22.173 INFO:teuthology.orchestra.run.smithi094.stdout: debug auth = 20 2024-03-29T11:18:22.173 INFO:teuthology.orchestra.run.smithi094.stdout: mon data avail warn = 5 2024-03-29T11:18:22.173 INFO:teuthology.orchestra.run.smithi094.stdout: mon mgr mkfs grace = 240 2024-03-29T11:18:22.173 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:18:22.174 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd reporter subtree level = osd 2024-03-29T11:18:22.174 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd prime pg temp = true 2024-03-29T11:18:22.174 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:18:22.174 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:18:22.174 INFO:teuthology.orchestra.run.smithi094.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:18:22.174 INFO:teuthology.orchestra.run.smithi094.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:18:22.175 INFO:teuthology.orchestra.run.smithi094.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:18:22.175 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:18:22.175 INFO:teuthology.orchestra.run.smithi094.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:18:22.175 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:18:22.175 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:18:22.175 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:18:22.175 INFO:teuthology.orchestra.run.smithi094.stdout: # 1m isn't quite enough 2024-03-29T11:18:22.176 INFO:teuthology.orchestra.run.smithi094.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:18:22.176 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:18:22.176 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:18:22.176 INFO:teuthology.orchestra.run.smithi094.stdout: mon min osdmap epochs = 50 2024-03-29T11:18:22.176 INFO:teuthology.orchestra.run.smithi094.stdout: mon osdmap full prune interval = 2 2024-03-29T11:18:22.176 INFO:teuthology.orchestra.run.smithi094.stdout: mon osdmap full prune min = 15 2024-03-29T11:18:22.177 INFO:teuthology.orchestra.run.smithi094.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:18:22.177 INFO:teuthology.orchestra.run.smithi094.stdout: mon scrub interval = 300 2024-03-29T11:18:22.177 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on pool no app = False 2024-03-29T11:18:22.177 INFO:teuthology.orchestra.run.smithi094.stdout: paxos service trim min = 10 2024-03-29T11:18:22.177 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T11:18:22.177 INFO:teuthology.orchestra.run.smithi094.stdout:[client] 2024-03-29T11:18:22.177 INFO:teuthology.orchestra.run.smithi094.stdout: rgw cache enabled = true 2024-03-29T11:18:22.177 INFO:teuthology.orchestra.run.smithi094.stdout: rgw enable ops log = true 2024-03-29T11:18:22.177 INFO:teuthology.orchestra.run.smithi094.stdout: rgw enable usage log = true 2024-03-29T11:18:22.177 INFO:teuthology.orchestra.run.smithi094.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:18:22.177 INFO:teuthology.orchestra.run.smithi094.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:18:22.178 INFO:teuthology.orchestra.run.smithi094.stdout: debug monc = 20 2024-03-29T11:18:22.178 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 1 2024-03-29T11:18:22.178 INFO:teuthology.orchestra.run.smithi094.stdout: debug objecter = 20 2024-03-29T11:18:22.178 INFO:teuthology.orchestra.run.smithi094.stdout: debug rados = 20 2024-03-29T11:18:22.178 INFO:teuthology.orchestra.run.smithi094.stdout:[mon.a] 2024-03-29T11:18:22.178 INFO:teuthology.orchestra.run.smithi094.stdout:[mon.c] 2024-03-29T11:18:22.178 INFO:teuthology.orchestra.run.smithi094.stdout:[mon.b] 2024-03-29T11:18:22.178 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T11:18:22.178 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T11:18:22.255 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T11:18:22.255 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-03-29T11:18:22.256 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T11:18:22.269 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-03-29T11:18:22.270 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T11:18:22.324 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi017.front.sepia.ceph.com 2024-03-29T11:18:22.324 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-03-29T11:18:22.324 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:18:22.324 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:18:22.403 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-03-29T11:18:22.403 DEBUG:teuthology.orchestra.run.smithi017:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:22.457 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi094.front.sepia.ceph.com 2024-03-29T11:18:22.457 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-29T11:18:22.457 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:18:22.457 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:18:22.496 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-29T11:18:22.497 DEBUG:teuthology.orchestra.run.smithi094:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:22.551 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T11:18:22.551 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T11:18:22.551 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:18:22.606 INFO:teuthology.orchestra.run.smithi017.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:18:22.609 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:18:22.672 INFO:teuthology.orchestra.run.smithi094.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:18:22.675 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T11:18:22.675 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T11:18:22.675 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T11:18:22.716 INFO:teuthology.orchestra.run.smithi017.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T11:18:22.732 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T11:18:22.774 INFO:teuthology.orchestra.run.smithi094.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T11:18:22.789 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T11:18:22.789 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi017.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi094.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:18:22.789 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T11:18:22.815 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:18:22.815 INFO:tasks.ceph:role: osd.0 2024-03-29T11:18:22.815 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi017.front.sepia.ceph.com 2024-03-29T11:18:22.815 DEBUG:teuthology.orchestra.run.smithi017:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:18:22.881 INFO:teuthology.orchestra.run.smithi017.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:18:22.882 INFO:teuthology.orchestra.run.smithi017.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:18:22.882 INFO:teuthology.orchestra.run.smithi017.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:18:22.882 INFO:teuthology.orchestra.run.smithi017.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:18:22.882 INFO:teuthology.orchestra.run.smithi017.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:18:22.882 INFO:teuthology.orchestra.run.smithi017.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:18:22.882 INFO:teuthology.orchestra.run.smithi017.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:18:22.882 INFO:teuthology.orchestra.run.smithi017.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:18:22.882 INFO:teuthology.orchestra.run.smithi017.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:18:22.882 INFO:teuthology.orchestra.run.smithi017.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:18:22.914 INFO:teuthology.orchestra.run.smithi017.stdout:Discarding blocks...Done. 2024-03-29T11:18:22.916 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi017.front.sepia.ceph.com -o noatime 2024-03-29T11:18:22.917 DEBUG:teuthology.orchestra.run.smithi017:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T11:18:22.987 DEBUG:teuthology.orchestra.run.smithi017:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T11:18:23.058 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T11:18:23.125 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:18:23.125 INFO:tasks.ceph:role: osd.1 2024-03-29T11:18:23.125 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi017.front.sepia.ceph.com 2024-03-29T11:18:23.125 DEBUG:teuthology.orchestra.run.smithi017:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:18:23.191 INFO:teuthology.orchestra.run.smithi017.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:18:23.191 INFO:teuthology.orchestra.run.smithi017.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:18:23.192 INFO:teuthology.orchestra.run.smithi017.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:18:23.192 INFO:teuthology.orchestra.run.smithi017.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:18:23.192 INFO:teuthology.orchestra.run.smithi017.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:18:23.192 INFO:teuthology.orchestra.run.smithi017.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:18:23.192 INFO:teuthology.orchestra.run.smithi017.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:18:23.192 INFO:teuthology.orchestra.run.smithi017.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:18:23.192 INFO:teuthology.orchestra.run.smithi017.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:18:23.192 INFO:teuthology.orchestra.run.smithi017.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:18:23.224 INFO:teuthology.orchestra.run.smithi017.stdout:Discarding blocks...Done. 2024-03-29T11:18:23.226 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi017.front.sepia.ceph.com -o noatime 2024-03-29T11:18:23.226 DEBUG:teuthology.orchestra.run.smithi017:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T11:18:23.297 DEBUG:teuthology.orchestra.run.smithi017:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T11:18:23.367 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T11:18:23.433 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:18:23.433 INFO:tasks.ceph:role: osd.2 2024-03-29T11:18:23.433 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi017.front.sepia.ceph.com 2024-03-29T11:18:23.433 DEBUG:teuthology.orchestra.run.smithi017:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:18:23.498 INFO:teuthology.orchestra.run.smithi017.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:18:23.498 INFO:teuthology.orchestra.run.smithi017.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:18:23.499 INFO:teuthology.orchestra.run.smithi017.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:18:23.499 INFO:teuthology.orchestra.run.smithi017.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:18:23.499 INFO:teuthology.orchestra.run.smithi017.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:18:23.499 INFO:teuthology.orchestra.run.smithi017.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:18:23.499 INFO:teuthology.orchestra.run.smithi017.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:18:23.499 INFO:teuthology.orchestra.run.smithi017.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:18:23.499 INFO:teuthology.orchestra.run.smithi017.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:18:23.499 INFO:teuthology.orchestra.run.smithi017.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:18:23.531 INFO:teuthology.orchestra.run.smithi017.stdout:Discarding blocks...Done. 2024-03-29T11:18:23.533 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi017.front.sepia.ceph.com -o noatime 2024-03-29T11:18:23.533 DEBUG:teuthology.orchestra.run.smithi017:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T11:18:23.603 DEBUG:teuthology.orchestra.run.smithi017:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T11:18:23.673 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T11:18:23.739 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:18:23.739 INFO:tasks.ceph:role: osd.3 2024-03-29T11:18:23.739 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi017.front.sepia.ceph.com 2024-03-29T11:18:23.740 DEBUG:teuthology.orchestra.run.smithi017:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:18:23.806 INFO:teuthology.orchestra.run.smithi017.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:18:23.806 INFO:teuthology.orchestra.run.smithi017.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:18:23.806 INFO:teuthology.orchestra.run.smithi017.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:18:23.806 INFO:teuthology.orchestra.run.smithi017.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:18:23.806 INFO:teuthology.orchestra.run.smithi017.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:18:23.806 INFO:teuthology.orchestra.run.smithi017.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:18:23.806 INFO:teuthology.orchestra.run.smithi017.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:18:23.806 INFO:teuthology.orchestra.run.smithi017.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:18:23.806 INFO:teuthology.orchestra.run.smithi017.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:18:23.806 INFO:teuthology.orchestra.run.smithi017.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:18:23.838 INFO:teuthology.orchestra.run.smithi017.stdout:Discarding blocks...Done. 2024-03-29T11:18:23.840 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi017.front.sepia.ceph.com -o noatime 2024-03-29T11:18:23.840 DEBUG:teuthology.orchestra.run.smithi017:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T11:18:23.910 DEBUG:teuthology.orchestra.run.smithi017:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T11:18:23.983 DEBUG:teuthology.orchestra.run.smithi017:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:24.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:24.107+0000 7f3db96d9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T11:18:24.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:24.107+0000 7f3db96d9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T11:18:24.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:24.107+0000 7f3db96d9640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:18:24.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:24.107+0000 7f3db96d9640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:18:24.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:24.107+0000 7f3db96d9640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:18:24.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:24.107+0000 7f3db96d9640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T11:18:26.925 DEBUG:teuthology.orchestra.run.smithi017:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T11:18:26.956 DEBUG:teuthology.orchestra.run.smithi017:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:27.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:27.070+0000 7f75c0a15640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T11:18:27.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:27.071+0000 7f75c0a15640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T11:18:27.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:27.071+0000 7f75c0a15640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:18:27.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:27.071+0000 7f75c0a15640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:18:27.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:27.071+0000 7f75c0a15640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:18:27.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:27.071+0000 7f75c0a15640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T11:18:29.894 DEBUG:teuthology.orchestra.run.smithi017:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T11:18:29.926 DEBUG:teuthology.orchestra.run.smithi017:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:30.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:30.039+0000 7fe9f158c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T11:18:30.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:30.040+0000 7fe9f158c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T11:18:30.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:30.040+0000 7fe9f158c640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:18:30.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:30.040+0000 7fe9f158c640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:18:30.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:30.040+0000 7fe9f158c640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:18:30.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:30.040+0000 7fe9f158c640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T11:18:32.864 DEBUG:teuthology.orchestra.run.smithi017:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T11:18:32.895 DEBUG:teuthology.orchestra.run.smithi017:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:32.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:33.008+0000 7f6ff72f7640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T11:18:32.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:33.009+0000 7f6ff72f7640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T11:18:32.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:33.009+0000 7f6ff72f7640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:18:32.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:33.009+0000 7f6ff72f7640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:18:32.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:33.009+0000 7f6ff72f7640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:18:32.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:33.009+0000 7f6ff72f7640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T11:18:35.830 DEBUG:teuthology.orchestra.run.smithi017:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T11:18:35.861 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T11:18:35.893 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:18:35.894 INFO:tasks.ceph:role: osd.4 2024-03-29T11:18:35.894 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi094.front.sepia.ceph.com 2024-03-29T11:18:35.894 DEBUG:teuthology.orchestra.run.smithi094:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:18:35.963 INFO:teuthology.orchestra.run.smithi094.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:18:35.963 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:18:35.963 INFO:teuthology.orchestra.run.smithi094.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:18:35.963 INFO:teuthology.orchestra.run.smithi094.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:18:35.964 INFO:teuthology.orchestra.run.smithi094.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:18:35.964 INFO:teuthology.orchestra.run.smithi094.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:18:35.964 INFO:teuthology.orchestra.run.smithi094.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:18:35.964 INFO:teuthology.orchestra.run.smithi094.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:18:35.964 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:18:35.964 INFO:teuthology.orchestra.run.smithi094.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:18:35.996 INFO:teuthology.orchestra.run.smithi094.stdout:Discarding blocks...Done. 2024-03-29T11:18:35.998 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi094.front.sepia.ceph.com -o noatime 2024-03-29T11:18:35.998 DEBUG:teuthology.orchestra.run.smithi094:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T11:18:36.071 DEBUG:teuthology.orchestra.run.smithi094:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T11:18:36.144 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T11:18:36.212 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:18:36.212 INFO:tasks.ceph:role: osd.5 2024-03-29T11:18:36.212 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi094.front.sepia.ceph.com 2024-03-29T11:18:36.212 DEBUG:teuthology.orchestra.run.smithi094:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:18:36.279 INFO:teuthology.orchestra.run.smithi094.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:18:36.279 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:18:36.279 INFO:teuthology.orchestra.run.smithi094.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:18:36.279 INFO:teuthology.orchestra.run.smithi094.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:18:36.279 INFO:teuthology.orchestra.run.smithi094.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:18:36.279 INFO:teuthology.orchestra.run.smithi094.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:18:36.279 INFO:teuthology.orchestra.run.smithi094.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:18:36.279 INFO:teuthology.orchestra.run.smithi094.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:18:36.279 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:18:36.279 INFO:teuthology.orchestra.run.smithi094.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:18:36.311 INFO:teuthology.orchestra.run.smithi094.stdout:Discarding blocks...Done. 2024-03-29T11:18:36.313 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi094.front.sepia.ceph.com -o noatime 2024-03-29T11:18:36.313 DEBUG:teuthology.orchestra.run.smithi094:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T11:18:36.389 DEBUG:teuthology.orchestra.run.smithi094:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T11:18:36.461 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T11:18:36.530 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:18:36.530 INFO:tasks.ceph:role: osd.6 2024-03-29T11:18:36.530 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi094.front.sepia.ceph.com 2024-03-29T11:18:36.530 DEBUG:teuthology.orchestra.run.smithi094:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:18:36.597 INFO:teuthology.orchestra.run.smithi094.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:18:36.598 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:18:36.598 INFO:teuthology.orchestra.run.smithi094.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:18:36.598 INFO:teuthology.orchestra.run.smithi094.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:18:36.598 INFO:teuthology.orchestra.run.smithi094.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:18:36.598 INFO:teuthology.orchestra.run.smithi094.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:18:36.598 INFO:teuthology.orchestra.run.smithi094.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:18:36.598 INFO:teuthology.orchestra.run.smithi094.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:18:36.598 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:18:36.598 INFO:teuthology.orchestra.run.smithi094.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:18:36.630 INFO:teuthology.orchestra.run.smithi094.stdout:Discarding blocks...Done. 2024-03-29T11:18:36.632 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi094.front.sepia.ceph.com -o noatime 2024-03-29T11:18:36.632 DEBUG:teuthology.orchestra.run.smithi094:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T11:18:36.703 DEBUG:teuthology.orchestra.run.smithi094:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T11:18:36.774 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T11:18:36.841 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:18:36.841 INFO:tasks.ceph:role: osd.7 2024-03-29T11:18:36.841 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi094.front.sepia.ceph.com 2024-03-29T11:18:36.842 DEBUG:teuthology.orchestra.run.smithi094:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:18:36.908 INFO:teuthology.orchestra.run.smithi094.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:18:36.908 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:18:36.908 INFO:teuthology.orchestra.run.smithi094.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:18:36.908 INFO:teuthology.orchestra.run.smithi094.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:18:36.908 INFO:teuthology.orchestra.run.smithi094.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:18:36.908 INFO:teuthology.orchestra.run.smithi094.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:18:36.908 INFO:teuthology.orchestra.run.smithi094.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:18:36.908 INFO:teuthology.orchestra.run.smithi094.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:18:36.908 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:18:36.908 INFO:teuthology.orchestra.run.smithi094.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:18:36.940 INFO:teuthology.orchestra.run.smithi094.stdout:Discarding blocks...Done. 2024-03-29T11:18:36.942 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi094.front.sepia.ceph.com -o noatime 2024-03-29T11:18:36.943 DEBUG:teuthology.orchestra.run.smithi094:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T11:18:37.019 DEBUG:teuthology.orchestra.run.smithi094:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T11:18:37.091 DEBUG:teuthology.orchestra.run.smithi094:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:37.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:37.215+0000 7f8d7cc48640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T11:18:37.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:37.215+0000 7f8d7cc48640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T11:18:37.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:37.215+0000 7f8d7cc48640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:18:37.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:37.215+0000 7f8d7cc48640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:18:37.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:37.215+0000 7f8d7cc48640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:18:37.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:37.216+0000 7f8d7cc48640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T11:18:40.038 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T11:18:40.068 DEBUG:teuthology.orchestra.run.smithi094:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:40.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:40.180+0000 7fa547a15640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T11:18:40.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:40.180+0000 7fa547a15640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T11:18:40.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:40.180+0000 7fa547a15640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:18:40.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:40.180+0000 7fa547a15640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:18:40.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:40.180+0000 7fa547a15640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:18:40.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:40.181+0000 7fa547a15640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T11:18:43.002 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T11:18:43.033 DEBUG:teuthology.orchestra.run.smithi094:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:43.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:43.146+0000 7f0121593640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T11:18:43.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:43.146+0000 7f0121593640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T11:18:43.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:43.146+0000 7f0121593640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:18:43.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:43.146+0000 7f0121593640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:18:43.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:43.146+0000 7f0121593640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:18:43.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:43.146+0000 7f0121593640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T11:18:45.960 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T11:18:45.989 DEBUG:teuthology.orchestra.run.smithi094:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:46.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:46.099+0000 7fc4dec48640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T11:18:46.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:46.099+0000 7fc4dec48640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T11:18:46.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:46.099+0000 7fc4dec48640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:18:46.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:46.099+0000 7fc4dec48640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:18:46.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:46.099+0000 7fc4dec48640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:18:46.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:46.099+0000 7fc4dec48640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T11:18:48.920 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T11:18:48.948 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T11:18:48.948 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-03-29T11:18:48.949 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T11:18:48.980 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-03-29T11:18:48.981 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T11:18:49.052 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-03-29T11:18:49.052 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T11:18:49.122 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-03-29T11:18:49.122 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T11:18:49.192 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-03-29T11:18:49.192 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T11:18:49.261 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-29T11:18:49.261 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T11:18:49.289 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-29T11:18:49.289 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T11:18:49.357 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-29T11:18:49.357 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T11:18:49.424 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-29T11:18:49.424 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T11:18:49.490 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-29T11:18:49.490 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T11:18:49.557 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-03-29T11:18:49.557 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T11:18:49.575 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-29T11:18:49.575 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T11:18:49.611 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T11:18:49.611 DEBUG:teuthology.orchestra.run.smithi017:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:18:49.617 DEBUG:teuthology.orchestra.run.smithi094:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:18:49.655 INFO:teuthology.orchestra.run.smithi017.stdout:[mgr.y] 2024-03-29T11:18:49.655 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQB+owZmTIubJRAAiy0ELE8UdfLMRD0vDJ7nyA== 2024-03-29T11:18:49.655 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.0] 2024-03-29T11:18:49.655 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQCAowZmI4BxBhAAyiuwCKE745GSNnQgTyo+Sw== 2024-03-29T11:18:49.655 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.1] 2024-03-29T11:18:49.655 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQCDowZmKLFHBBAAFjAYIdiiKiAPKrALrD+onA== 2024-03-29T11:18:49.655 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.2] 2024-03-29T11:18:49.656 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQCGowZmcUJwAhAAKYLgOahwQCAfOhYv1fydzA== 2024-03-29T11:18:49.656 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.3] 2024-03-29T11:18:49.656 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQCJowZmnf6VABAA2V+ED2NxW+tCdR0b47J4KQ== 2024-03-29T11:18:49.656 INFO:teuthology.orchestra.run.smithi017.stdout:[mgr.x] 2024-03-29T11:18:49.656 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQB+owZmYdCIKRAALmqbAI9WuefWE8AxdcZoAg== 2024-03-29T11:18:49.656 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.4] 2024-03-29T11:18:49.656 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQCNowZmzvz4DBAAStyV1dXmMxkUQpSF8GQQ6Q== 2024-03-29T11:18:49.656 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.5] 2024-03-29T11:18:49.656 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQCQowZm6cnRChAAsJ/FtRFZYYzSU1Xk+LYzlw== 2024-03-29T11:18:49.656 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.6] 2024-03-29T11:18:49.656 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQCTowZmYrHICBAAlLwH2xBrbkrwgd95JAVlmA== 2024-03-29T11:18:49.656 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.7] 2024-03-29T11:18:49.656 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQCWowZmmpD6BRAAxtWgAyrmr2bReV+dpxsqbw== 2024-03-29T11:18:49.656 INFO:teuthology.orchestra.run.smithi017.stdout:[client.0] 2024-03-29T11:18:49.657 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQB+owZmMAQtLBAAJcCYyMVl48FUNY2vbprj7Q== 2024-03-29T11:18:49.657 INFO:teuthology.orchestra.run.smithi017.stdout:[client.1] 2024-03-29T11:18:49.657 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQB+owZm/c6ULxAA3JJrX/djEhBP64DTKJ4VkQ== 2024-03-29T11:18:49.676 INFO:teuthology.orchestra.run.smithi094.stdout:[mgr.y] 2024-03-29T11:18:49.677 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQB+owZmTIubJRAAiy0ELE8UdfLMRD0vDJ7nyA== 2024-03-29T11:18:49.677 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.0] 2024-03-29T11:18:49.677 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQCAowZmI4BxBhAAyiuwCKE745GSNnQgTyo+Sw== 2024-03-29T11:18:49.677 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.1] 2024-03-29T11:18:49.677 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQCDowZmKLFHBBAAFjAYIdiiKiAPKrALrD+onA== 2024-03-29T11:18:49.677 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.2] 2024-03-29T11:18:49.677 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQCGowZmcUJwAhAAKYLgOahwQCAfOhYv1fydzA== 2024-03-29T11:18:49.677 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.3] 2024-03-29T11:18:49.677 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQCJowZmnf6VABAA2V+ED2NxW+tCdR0b47J4KQ== 2024-03-29T11:18:49.677 INFO:teuthology.orchestra.run.smithi094.stdout:[mgr.x] 2024-03-29T11:18:49.677 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQB+owZmYdCIKRAALmqbAI9WuefWE8AxdcZoAg== 2024-03-29T11:18:49.677 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.4] 2024-03-29T11:18:49.677 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQCNowZmzvz4DBAAStyV1dXmMxkUQpSF8GQQ6Q== 2024-03-29T11:18:49.677 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.5] 2024-03-29T11:18:49.678 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQCQowZm6cnRChAAsJ/FtRFZYYzSU1Xk+LYzlw== 2024-03-29T11:18:49.678 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.6] 2024-03-29T11:18:49.678 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQCTowZmYrHICBAAlLwH2xBrbkrwgd95JAVlmA== 2024-03-29T11:18:49.678 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.7] 2024-03-29T11:18:49.678 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQCWowZmmpD6BRAAxtWgAyrmr2bReV+dpxsqbw== 2024-03-29T11:18:49.678 INFO:teuthology.orchestra.run.smithi094.stdout:[client.0] 2024-03-29T11:18:49.678 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQB+owZmMAQtLBAAJcCYyMVl48FUNY2vbprj7Q== 2024-03-29T11:18:49.678 INFO:teuthology.orchestra.run.smithi094.stdout:[client.1] 2024-03-29T11:18:49.678 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQB+owZm/c6ULxAA3JJrX/djEhBP64DTKJ4VkQ== 2024-03-29T11:18:49.679 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:18:49.700 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:18:49.771 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:18:49.774 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:18:49.827 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:18:49.830 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:18:49.882 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:18:49.884 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:18:49.936 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:18:49.938 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:18:49.989 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:18:49.992 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:18:50.043 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:18:50.045 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:18:50.097 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:18:50.099 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:18:50.150 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:18:50.152 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:18:50.204 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:18:50.206 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:18:50.257 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:18:50.260 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:18:50.314 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:18:50.316 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:18:50.370 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T11:18:50.370 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T11:18:50.397 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:18:50.493 DEBUG:teuthology.orchestra.run.smithi017:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T11:18:50.519 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T11:18:50.586 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:18:50.683 DEBUG:teuthology.orchestra.run.smithi017:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T11:18:50.709 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T11:18:50.737 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:18:50.836 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T11:18:50.863 DEBUG:teuthology.orchestra.run.smithi017:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:50.865 DEBUG:teuthology.orchestra.run.smithi094:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:50.917 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T11:18:50.918 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T11:18:50.918 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T11:18:50.921 INFO:tasks.ceph.mon.a:Started 2024-03-29T11:18:50.921 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T11:18:50.921 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T11:18:50.923 INFO:tasks.ceph.mon.c:Started 2024-03-29T11:18:50.923 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T11:18:50.923 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T11:18:50.960 INFO:tasks.ceph.mon.b:Started 2024-03-29T11:18:50.960 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T11:18:50.961 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T11:18:50.961 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T11:18:50.963 INFO:tasks.ceph.mgr.y:Started 2024-03-29T11:18:50.963 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T11:18:50.963 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T11:18:50.966 INFO:tasks.ceph.mgr.x:Started 2024-03-29T11:18:50.966 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T11:18:50.966 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T11:18:51.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:51.092+0000 7fb401c93640 10 monclient: get_monmap_and_config 2024-03-29T11:18:51.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:51.092+0000 7fb401c93640 10 monclient: build_initial_monmap 2024-03-29T11:18:51.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:51.092+0000 7fb401c93640 10 monclient: monmap: 2024-03-29T11:18:51.069 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-03-29T11:18:51.069 INFO:teuthology.orchestra.run.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:51.069 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-03-29T11:18:51.093011+0000 2024-03-29T11:18:51.069 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-03-29T11:18:51.093011+0000 2024-03-29T11:18:51.070 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:51.070 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-03-29T11:18:51.070 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:18:51.070 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:18:51.070 INFO:teuthology.orchestra.run.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:18:51.070 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-03-29T11:18:51.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:51.092+0000 7fb401c93640 1 Processor -- start 2024-03-29T11:18:51.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:51.092+0000 7fb401c93640 1 -- start start 2024-03-29T11:18:51.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:51.092+0000 7fb401c93640 10 monclient: init 2024-03-29T11:18:51.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:51.093+0000 7fb401c93640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:51.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:51.093+0000 7fb401c93640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:18:51.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:51.093+0000 7fb401c93640 10 monclient(hunting): picked mon.noname-a con 0x7fb3fc066920 addr v1:172.21.15.17:6789/0 2024-03-29T11:18:51.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:51.093+0000 7fb401c93640 10 monclient(hunting): picked mon.noname-b con 0x7fb3fc066d00 addr v1:172.21.15.17:6790/0 2024-03-29T11:18:51.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:51.093+0000 7fb401c93640 10 monclient(hunting): picked mon.noname-c con 0x7fb3fc0af280 addr v1:172.21.15.94:6789/0 2024-03-29T11:18:51.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:51.093+0000 7fb401c93640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3fc0afd20 con 0x7fb3fc066920 2024-03-29T11:18:51.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:51.093+0000 7fb401c93640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3fc13fba0 con 0x7fb3fc0af280 2024-03-29T11:18:51.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:51.093+0000 7fb401c93640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3fc0b0f40 con 0x7fb3fc066d00 2024-03-29T11:18:51.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:51.093+0000 7fb401c93640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:51.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:51.093+0000 7fb401c93640 10 monclient(hunting): authenticate will time out at 837.298217s 2024-03-29T11:18:51.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:51.093+0000 7fb3fb7fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fb3fc066920 0x7fb3fc145540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:34958/0 (socket says 172.21.15.17:34958) 2024-03-29T11:18:51.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:51.093+0000 7fb3fb7fe640 1 -- 172.21.15.17:0/2788961774 learned_addr learned my addr 172.21.15.17:0/2788961774 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:18:54.070 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:54.093+0000 7fb3f97fa640 10 monclient(hunting): tick 2024-03-29T11:18:54.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:54.093+0000 7fb3f97fa640 10 monclient(hunting): _check_auth_tickets 2024-03-29T11:18:54.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:54.093+0000 7fb3f97fa640 1 monclient(hunting): continuing hunt 2024-03-29T11:18:54.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:54.093+0000 7fb3f97fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-29T11:18:54.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:54.093+0000 7fb3f97fa640 1 -- 172.21.15.17:0/2788961774 >> v1:172.21.15.17:6790/0 conn(0x7fb3fc066d00 legacy=0x7fb3fc072390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:54.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:54.093+0000 7fb3f97fa640 1 -- 172.21.15.17:0/2788961774 >> v1:172.21.15.94:6789/0 conn(0x7fb3fc0af280 legacy=0x7fb3fc071750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:54.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:54.093+0000 7fb3f97fa640 1 -- 172.21.15.17:0/2788961774 >> v1:172.21.15.17:6789/0 conn(0x7fb3fc066920 legacy=0x7fb3fc145540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:54.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:54.093+0000 7fb3f97fa640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:18:54.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:54.093+0000 7fb3f97fa640 10 monclient(hunting): picked mon.noname-c con 0x7fb3e80036e0 addr v1:172.21.15.94:6789/0 2024-03-29T11:18:54.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:54.093+0000 7fb3f97fa640 10 monclient(hunting): picked mon.noname-b con 0x7fb3e80072d0 addr v1:172.21.15.17:6790/0 2024-03-29T11:18:54.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:54.093+0000 7fb3f97fa640 10 monclient(hunting): picked mon.noname-a con 0x7fb3e800b000 addr v1:172.21.15.17:6789/0 2024-03-29T11:18:54.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:54.093+0000 7fb3f97fa640 1 -- 172.21.15.17:0/2788961774 --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3fc0b10d0 con 0x7fb3e800b000 2024-03-29T11:18:54.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:54.093+0000 7fb3f97fa640 1 -- 172.21.15.17:0/2788961774 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3e800fdf0 con 0x7fb3e80036e0 2024-03-29T11:18:54.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:54.093+0000 7fb3f97fa640 1 -- 172.21.15.17:0/2788961774 --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3e8011030 con 0x7fb3e80072d0 2024-03-29T11:18:54.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:54.093+0000 7fb3f97fa640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:56.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.258+0000 7fb3fa7fc640 1 -- 172.21.15.17:0/2788961774 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1314564439 0 0) 0x7fb3fc0b10d0 con 0x7fb3e800b000 2024-03-29T11:18:56.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.258+0000 7fb3fa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:18:56.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.258+0000 7fb3fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:56.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.258+0000 7fb3fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:56.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.258+0000 7fb3fa7fc640 10 monclient(hunting): my global_id is 4101 2024-03-29T11:18:56.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.258+0000 7fb3fa7fc640 1 -- 172.21.15.17:0/2788961774 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3e4003520 con 0x7fb3e800b000 2024-03-29T11:18:56.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.258+0000 7fb3fa7fc640 1 -- 172.21.15.17:0/2788961774 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1589740013 0 0) 0x7fb3e800fdf0 con 0x7fb3e80036e0 2024-03-29T11:18:56.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.258+0000 7fb3fa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:18:56.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.258+0000 7fb3fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:56.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.258+0000 7fb3fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:56.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.258+0000 7fb3fa7fc640 10 monclient(hunting): my global_id is 4105 2024-03-29T11:18:56.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.258+0000 7fb3fa7fc640 1 -- 172.21.15.17:0/2788961774 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3fc0b10d0 con 0x7fb3e80036e0 2024-03-29T11:18:56.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.258+0000 7fb3fb7fe640 1 -- 172.21.15.17:0/2788961774 >> v1:172.21.15.94:6789/0 conn(0x7fb3e80036e0 legacy=0x7fb3e8005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-03-29T11:18:56.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.258+0000 7fb3fb7fe640 1 -- 172.21.15.17:0/2788961774 >> v1:172.21.15.94:6789/0 conn(0x7fb3e80036e0 legacy=0x7fb3e8005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:18:56.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.258+0000 7fb3fb7fe640 1 --1- 172.21.15.17:0/2788961774 >> v1:172.21.15.94:6789/0 conn(0x7fb3e80036e0 0x7fb3e8005b30 :-1 s=OPENED pgs=11 cs=1 l=1).handle_message read tag failed 2024-03-29T11:18:56.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.258+0000 7fb3fb7fe640 1 --1- 172.21.15.17:0/2788961774 >> v1:172.21.15.94:6789/0 conn(0x7fb3e80036e0 0x7fb3e8005b30 :-1 s=OPENED pgs=11 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:18:56.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.258+0000 7fb3fa7fc640 1 -- 172.21.15.17:0/2788961774 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2881037854 0 0) 0x7fb3e8011030 con 0x7fb3e80072d0 2024-03-29T11:18:56.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.258+0000 7fb3fa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:18:56.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.258+0000 7fb3fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:56.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.258+0000 7fb3fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:56.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.258+0000 7fb3fa7fc640 10 monclient(hunting): my global_id is 4106 2024-03-29T11:18:56.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.258+0000 7fb3fa7fc640 1 -- 172.21.15.17:0/2788961774 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3e800fdf0 con 0x7fb3e80072d0 2024-03-29T11:18:56.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.258+0000 7fb3fa7fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.94:6789/0 2024-03-29T11:18:56.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.258+0000 7fb3faffd640 1 -- 172.21.15.17:0/2788961774 >> v1:172.21.15.17:6790/0 conn(0x7fb3e80072d0 legacy=0x7fb3e8009760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-29T11:18:56.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.258+0000 7fb3faffd640 1 -- 172.21.15.17:0/2788961774 >> v1:172.21.15.17:6790/0 conn(0x7fb3e80072d0 legacy=0x7fb3e8009760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:18:56.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.258+0000 7fb3faffd640 1 --1- 172.21.15.17:0/2788961774 >> v1:172.21.15.17:6790/0 conn(0x7fb3e80072d0 0x7fb3e8009760 :-1 s=OPENED pgs=11 cs=1 l=1).handle_message read tag failed 2024-03-29T11:18:56.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.258+0000 7fb3faffd640 1 --1- 172.21.15.17:0/2788961774 >> v1:172.21.15.17:6790/0 conn(0x7fb3e80072d0 0x7fb3e8009760 :-1 s=OPENED pgs=11 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:18:56.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.258+0000 7fb3fa7fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.17:6790/0 2024-03-29T11:18:56.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.259+0000 7fb3fa7fc640 1 -- 172.21.15.17:0/2788961774 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3252705389 0 0) 0x7fb3e4003520 con 0x7fb3e800b000 2024-03-29T11:18:56.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.259+0000 7fb3fa7fc640 1 -- 172.21.15.17:0/2788961774 >> v1:172.21.15.17:6790/0 conn(0x7fb3e80072d0 legacy=0x7fb3e8009760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:18:56.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.259+0000 7fb3fa7fc640 1 -- 172.21.15.17:0/2788961774 >> v1:172.21.15.94:6789/0 conn(0x7fb3e80036e0 legacy=0x7fb3e8005b30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:18:56.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.259+0000 7fb3fa7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:18:56.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.259+0000 7fb3fa7fc640 1 monclient: found mon.noname-a 2024-03-29T11:18:56.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.259+0000 7fb3fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:56.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.259+0000 7fb3fa7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.17:6789/0 2024-03-29T11:18:56.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.259+0000 7fb3fa7fc640 1 -- 172.21.15.17:0/2788961774 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3e80122d0 con 0x7fb3e800b000 2024-03-29T11:18:56.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.259+0000 7fb3fa7fc640 10 monclient: _finish_auth 0 2024-03-29T11:18:56.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.259+0000 7fb3fa7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:18:56.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.259+0000 7fb3fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:56.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.259+0000 7fb3fa7fc640 1 -- 172.21.15.17:0/2788961774 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fb3f4001a50 con 0x7fb3e800b000 2024-03-29T11:18:56.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.259+0000 7fb401c93640 5 monclient: authenticate success, global_id 4101 2024-03-29T11:18:56.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.259+0000 7fb401c93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:56.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.259+0000 7fb3fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:56.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.259+0000 7fb3fa7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:18:56.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.259+0000 7fb3fa7fc640 10 monclient: dump: 2024-03-29T11:18:56.240 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-03-29T11:18:56.240 INFO:teuthology.orchestra.run.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:56.240 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:56.240 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:56.240 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:56.240 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-03-29T11:18:56.240 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:56.240 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:56.240 INFO:teuthology.orchestra.run.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:56.240 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-03-29T11:18:56.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.259+0000 7fb401c93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:56.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.261+0000 7fb3fa7fc640 1 -- 172.21.15.17:0/2788961774 <== mon.0 v1:172.21.15.17:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb3f4001bb0 con 0x7fb3e800b000 2024-03-29T11:18:56.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.261+0000 7fb3fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:56.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.261+0000 7fb3fa7fc640 1 -- 172.21.15.17:0/2788961774 <== mon.0 v1:172.21.15.17:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fb3f4001fc0 con 0x7fb3e800b000 2024-03-29T11:18:56.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.261+0000 7fb3fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:56.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.261+0000 7fb3fa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:56.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.261+0000 7fb3fa7fc640 10 monclient: dump: 2024-03-29T11:18:56.244 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-03-29T11:18:56.244 INFO:teuthology.orchestra.run.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:56.244 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:56.244 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:56.244 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:56.244 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-03-29T11:18:56.245 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:56.245 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:56.245 INFO:teuthology.orchestra.run.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:56.245 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-03-29T11:18:56.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.261+0000 7fb401c93640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:56.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.261+0000 7fb401c93640 4 set_mon_vals no callback set 2024-03-29T11:18:56.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.262+0000 7fb401c93640 10 monclient: shutdown 2024-03-29T11:18:56.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.262+0000 7fb401c93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:56.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.262+0000 7fb401c93640 1 -- 172.21.15.17:0/2788961774 >> v1:172.21.15.17:6789/0 conn(0x7fb3e800b000 legacy=0x7fb3e800d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:56.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.262+0000 7fb401c93640 1 -- 172.21.15.17:0/2788961774 shutdown_connections 2024-03-29T11:18:56.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.262+0000 7fb401c93640 1 -- 172.21.15.17:0/2788961774 >> 172.21.15.17:0/2788961774 conn(0x7fb3fc0b4760 msgr2=0x7fb3fc0b4b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:56.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.262+0000 7fb401c93640 1 -- 172.21.15.17:0/2788961774 shutdown_connections 2024-03-29T11:18:56.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.262+0000 7fb401c93640 1 -- 172.21.15.17:0/2788961774 wait complete. 2024-03-29T11:18:56.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.263+0000 7fb401c93640 10 monclient: build_initial_monmap 2024-03-29T11:18:56.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.263+0000 7fb401c93640 10 monclient: monmap: 2024-03-29T11:18:56.246 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-03-29T11:18:56.246 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:56.246 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-03-29T11:18:56.246 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-03-29T11:18:56.246 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:56.246 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-03-29T11:18:56.246 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:18:56.246 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:18:56.246 INFO:teuthology.orchestra.run.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:18:56.246 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-03-29T11:18:56.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.264+0000 7fb401c93640 1 librados: starting msgr at 2024-03-29T11:18:56.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.264+0000 7fb401c93640 1 librados: starting objecter 2024-03-29T11:18:56.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.264+0000 7fb401c93640 1 Processor -- start 2024-03-29T11:18:56.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.264+0000 7fb401c93640 1 -- start start 2024-03-29T11:18:56.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.264+0000 7fb401c93640 1 librados: setting wanted keys 2024-03-29T11:18:56.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.264+0000 7fb401c93640 1 librados: calling monclient init 2024-03-29T11:18:56.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.264+0000 7fb401c93640 10 monclient: init 2024-03-29T11:18:56.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.265+0000 7fb401c93640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:56.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.265+0000 7fb401c93640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:18:56.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.265+0000 7fb401c93640 10 monclient(hunting): picked mon.noname-b con 0x7fb3e80036e0 addr v1:172.21.15.94:6789/0 2024-03-29T11:18:56.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.265+0000 7fb401c93640 10 monclient(hunting): picked mon.noname-a con 0x7fb3e80072d0 addr v1:172.21.15.17:6789/0 2024-03-29T11:18:56.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.266+0000 7fb3fb7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fb3e80036e0 0x7fb3fc1d77b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:44528/0 (socket says 172.21.15.17:44528) 2024-03-29T11:18:56.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.266+0000 7fb3faffd640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fb3e80072d0 0x7fb3fc13a520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:35018/0 (socket says 172.21.15.17:35018) 2024-03-29T11:18:56.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.266+0000 7fb3fb7fe640 1 -- 172.21.15.17:0/690825745 learned_addr learned my addr 172.21.15.17:0/690825745 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:18:56.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.266+0000 7fb401c93640 10 monclient(hunting): picked mon.noname-c con 0x7fb3e800b000 addr v1:172.21.15.17:6790/0 2024-03-29T11:18:56.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.266+0000 7fb401c93640 1 -- 172.21.15.17:0/690825745 --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3fc1d7320 con 0x7fb3e80072d0 2024-03-29T11:18:56.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.266+0000 7fb401c93640 1 -- 172.21.15.17:0/690825745 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3fc142f80 con 0x7fb3e80036e0 2024-03-29T11:18:56.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.266+0000 7fb3fbfff640 1 -- 172.21.15.17:0/690825745 >> v1:172.21.15.17:6790/0 conn(0x7fb3e800b000 legacy=0x7fb3fc13ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-29T11:18:56.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.266+0000 7fb3fbfff640 1 -- 172.21.15.17:0/690825745 >> v1:172.21.15.17:6790/0 conn(0x7fb3e800b000 legacy=0x7fb3fc13ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:18:56.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.266+0000 7fb3fbfff640 1 --1- 172.21.15.17:0/690825745 >> v1:172.21.15.17:6790/0 conn(0x7fb3e800b000 0x7fb3fc13ac30 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-03-29T11:18:56.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.266+0000 7fb401c93640 1 -- 172.21.15.17:0/690825745 --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3fc13b340 con 0x7fb3e800b000 2024-03-29T11:18:56.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.266+0000 7fb3fbfff640 1 --1- 172.21.15.17:0/690825745 >> v1:172.21.15.17:6790/0 conn(0x7fb3e800b000 0x7fb3fc13ac30 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-03-29T11:18:56.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.266+0000 7fb401c93640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:56.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.266+0000 7fb401c93640 10 monclient(hunting): authenticate will time out at 842.471234s 2024-03-29T11:18:56.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.266+0000 7fb3e3fff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.17:6790/0 2024-03-29T11:18:56.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.266+0000 7fb3e3fff640 10 client.?.objecter ms_handle_connect 0x7fb3e80036e0 2024-03-29T11:18:56.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.266+0000 7fb3e3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:56.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.266+0000 7fb3e3fff640 10 client.?.objecter ms_handle_connect 0x7fb3e80072d0 2024-03-29T11:18:56.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.266+0000 7fb3e3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:56.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.266+0000 7fb3e3fff640 1 -- 172.21.15.17:0/690825745 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 49548811 0 0) 0x7fb3fc1d7320 con 0x7fb3e80072d0 2024-03-29T11:18:56.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.266+0000 7fb3e3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:18:56.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.266+0000 7fb3e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:56.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.266+0000 7fb3e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:56.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.266+0000 7fb3e3fff640 10 monclient(hunting): my global_id is 4110 2024-03-29T11:18:56.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.266+0000 7fb3e3fff640 1 -- 172.21.15.17:0/690825745 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3e8010740 con 0x7fb3e80072d0 2024-03-29T11:18:56.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.268+0000 7fb3e3fff640 1 -- 172.21.15.17:0/690825745 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3363483733 0 0) 0x7fb3fc142f80 con 0x7fb3e80036e0 2024-03-29T11:18:56.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.268+0000 7fb3e3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:18:56.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.268+0000 7fb3e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:56.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.268+0000 7fb3e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:56.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.268+0000 7fb3e3fff640 10 monclient(hunting): my global_id is 4114 2024-03-29T11:18:56.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.268+0000 7fb3e3fff640 1 -- 172.21.15.17:0/690825745 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3fc1d7320 con 0x7fb3e80036e0 2024-03-29T11:18:56.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.268+0000 7fb3e3fff640 1 -- 172.21.15.17:0/690825745 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1909044080 0 0) 0x7fb3fc1d7320 con 0x7fb3e80036e0 2024-03-29T11:18:56.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.268+0000 7fb3e3fff640 1 -- 172.21.15.17:0/690825745 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb3fc142f80 con 0x7fb3e80036e0 2024-03-29T11:18:56.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.268+0000 7fb3e3fff640 1 -- 172.21.15.17:0/690825745 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fb3ec001970 con 0x7fb3e80036e0 2024-03-29T11:18:56.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.268+0000 7fb3e3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:56.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.268+0000 7fb3e3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:18:56.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.268+0000 7fb3e3fff640 10 monclient(hunting): dump: 2024-03-29T11:18:56.252 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-03-29T11:18:56.252 INFO:teuthology.orchestra.run.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:56.252 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:56.252 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:56.252 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:56.252 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-03-29T11:18:56.252 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:56.252 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:56.252 INFO:teuthology.orchestra.run.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:56.252 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-03-29T11:18:56.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.268+0000 7fb3e3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:56.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.268+0000 7fb3e3fff640 1 -- 172.21.15.17:0/690825745 <== mon.1 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2567746776 0 0) 0x7fb3fc142f80 con 0x7fb3e80036e0 2024-03-29T11:18:56.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.268+0000 7fb3e3fff640 1 -- 172.21.15.17:0/690825745 >> v1:172.21.15.17:6790/0 conn(0x7fb3e800b000 legacy=0x7fb3fc13ac30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:18:56.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.268+0000 7fb3e3fff640 1 -- 172.21.15.17:0/690825745 >> v1:172.21.15.17:6789/0 conn(0x7fb3e80072d0 legacy=0x7fb3fc13a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:56.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.268+0000 7fb3e3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:18:56.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.268+0000 7fb3e3fff640 1 monclient: found mon.b 2024-03-29T11:18:56.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.268+0000 7fb3e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:56.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.268+0000 7fb3e3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:18:56.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.268+0000 7fb3e3fff640 1 -- 172.21.15.17:0/690825745 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3fc140b90 con 0x7fb3e80036e0 2024-03-29T11:18:56.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.268+0000 7fb3e3fff640 10 monclient: _finish_auth 0 2024-03-29T11:18:56.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.268+0000 7fb3e3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:18:56.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.268+0000 7fb3e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:56.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.268+0000 7fb401c93640 5 monclient: authenticate success, global_id 4114 2024-03-29T11:18:56.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.268+0000 7fb401c93640 10 monclient: _renew_subs 2024-03-29T11:18:56.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.268+0000 7fb401c93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:18:56.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.268+0000 7fb401c93640 1 -- 172.21.15.17:0/690825745 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3fc1d7ec0 con 0x7fb3e80036e0 2024-03-29T11:18:56.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.268+0000 7fb401c93640 10 client.4114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:56.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.268+0000 7fb401c93640 10 monclient: _renew_subs 2024-03-29T11:18:56.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.268+0000 7fb401c93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:18:56.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.268+0000 7fb401c93640 1 -- 172.21.15.17:0/690825745 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3fc142010 con 0x7fb3e80036e0 2024-03-29T11:18:56.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.268+0000 7fb401c93640 1 librados: init done 2024-03-29T11:18:56.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.269+0000 7fb3e3fff640 1 -- 172.21.15.17:0/690825745 <== mon.1 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb3ec0047e0 con 0x7fb3e80036e0 2024-03-29T11:18:56.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.269+0000 7fb3e3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:56.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.269+0000 7fb3e3fff640 1 -- 172.21.15.17:0/690825745 <== mon.1 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fb3ec004af0 con 0x7fb3e80036e0 2024-03-29T11:18:56.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.269+0000 7fb3e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:56.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.269+0000 7fb3e3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:18:56.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.269+0000 7fb3e3fff640 10 monclient: dump: 2024-03-29T11:18:56.254 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-03-29T11:18:56.254 INFO:teuthology.orchestra.run.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:56.254 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:56.255 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:56.255 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:56.255 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-03-29T11:18:56.255 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:56.255 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:56.255 INFO:teuthology.orchestra.run.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:56.255 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-03-29T11:18:56.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.269+0000 7fb3e3fff640 1 -- 172.21.15.17:0/690825745 <== mon.1 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb3ec0050a0 con 0x7fb3e80036e0 2024-03-29T11:18:56.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.269+0000 7fb3e3fff640 1 -- 172.21.15.17:0/690825745 <== mon.1 v1:172.21.15.94:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 416851505 0 0) 0x7fb3ec006620 con 0x7fb3e80036e0 2024-03-29T11:18:56.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.269+0000 7fb3e3fff640 10 client.4114.objecter ms_dispatch 0x7fb3fc000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T11:18:56.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.269+0000 7fb3e3fff640 3 client.4114.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T11:18:56.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.269+0000 7fb3e3fff640 3 client.4114.objecter handle_osd_map decoding full epoch 1 2024-03-29T11:18:56.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.269+0000 7fb3f97fa640 4 set_mon_vals no callback set 2024-03-29T11:18:56.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.269+0000 7fb3e3fff640 20 client.4114.objecter dump_active .. 0 homeless 2024-03-29T11:18:56.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.270+0000 7fb401c93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:56.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.270+0000 7fb401c93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:56.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.270+0000 7fb401c93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:18:56.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.270+0000 7fb401c93640 1 -- 172.21.15.17:0/690825745 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3fc1d70f0 con 0x7fb3e80036e0 2024-03-29T11:18:56.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.275+0000 7fb3e3fff640 1 -- 172.21.15.17:0/690825745 <== mon.1 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb3fc1d70f0 con 0x7fb3e80036e0 2024-03-29T11:18:56.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.275+0000 7fb3e3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:56.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.275+0000 7fb3e3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:56.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.362+0000 7fb401c93640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:18:56.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.362+0000 7fb401c93640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:18:56.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.362+0000 7fb401c93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:18:56.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.362+0000 7fb401c93640 1 -- 172.21.15.17:0/690825745 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fb3fc04f730 con 0x7fb3e80036e0 2024-03-29T11:18:56.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.365+0000 7fb3e3fff640 1 -- 172.21.15.17:0/690825745 <== mon.1 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7fb3ec005610 con 0x7fb3e80036e0 2024-03-29T11:18:56.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.365+0000 7fb3e3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:18:56.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.365+0000 7fb3e3fff640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T11:18:56.342 INFO:teuthology.orchestra.run.smithi017.stderr:adjusted tunables profile to default 2024-03-29T11:18:56.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.366+0000 7fb401c93640 10 librados: watch_flush enter 2024-03-29T11:18:56.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.366+0000 7fb401c93640 10 librados: watch_flush exit 2024-03-29T11:18:56.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.366+0000 7fb401c93640 20 client.4114.objecter shutdown clearing up homeless session... 2024-03-29T11:18:56.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.366+0000 7fb401c93640 10 client.4114.objecter successfully canceled tick 2024-03-29T11:18:56.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.366+0000 7fb401c93640 10 monclient: shutdown 2024-03-29T11:18:56.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.366+0000 7fb401c93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:56.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.366+0000 7fb401c93640 1 -- 172.21.15.17:0/690825745 >> v1:172.21.15.94:6789/0 conn(0x7fb3e80036e0 legacy=0x7fb3fc1d77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:56.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.366+0000 7fb401c93640 1 -- 172.21.15.17:0/690825745 shutdown_connections 2024-03-29T11:18:56.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.366+0000 7fb401c93640 1 -- 172.21.15.17:0/690825745 >> 172.21.15.17:0/690825745 conn(0x7fb3fc0b4760 msgr2=0x7fb3fc06f010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:56.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.366+0000 7fb401c93640 1 -- 172.21.15.17:0/690825745 shutdown_connections 2024-03-29T11:18:56.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.366+0000 7fb401c93640 1 -- 172.21.15.17:0/690825745 wait complete. 2024-03-29T11:18:56.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:56.366+0000 7fb401c93640 1 librados: shutdown 2024-03-29T11:18:56.358 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T11:18:56.358 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T11:18:56.358 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-03-29T11:18:56.358 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T11:18:56.385 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-03-29T11:18:56.385 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T11:18:56.451 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-03-29T11:18:56.451 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T11:18:56.518 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-03-29T11:18:56.518 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T11:18:56.585 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-29T11:18:56.585 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T11:18:56.612 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-29T11:18:56.612 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T11:18:56.679 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-29T11:18:56.680 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T11:18:56.746 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-29T11:18:56.746 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T11:18:56.749 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-03-29T11:18:56.772+0000 7fcdfee3c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:18:56.758 INFO:tasks.ceph.mgr.y.smithi017.stderr:2024-03-29T11:18:56.781+0000 7f6f3d22e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:18:56.775 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster ceph osd new e7583f99-c8fc-47b7-98c9-000b5892563e 0 2024-03-29T11:18:56.805 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-03-29T11:18:56.829+0000 7fcdfee3c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:18:56.813 INFO:tasks.ceph.mgr.y.smithi017.stderr:2024-03-29T11:18:56.837+0000 7f6f3d22e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:18:56.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.928+0000 7f36352c1640 10 monclient: get_monmap_and_config 2024-03-29T11:18:56.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.928+0000 7f36352c1640 10 monclient: build_initial_monmap 2024-03-29T11:18:56.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.928+0000 7f36352c1640 10 monclient: monmap: 2024-03-29T11:18:56.905 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-03-29T11:18:56.905 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:56.905 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:56.929187+0000 2024-03-29T11:18:56.905 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:56.929187+0000 2024-03-29T11:18:56.905 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:56.905 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:56.905 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:18:56.905 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:18:56.905 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:18:56.906 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:56.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.928+0000 7f36352c1640 1 Processor -- start 2024-03-29T11:18:56.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.928+0000 7f36352c1640 1 -- start start 2024-03-29T11:18:56.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.928+0000 7f36352c1640 10 monclient: init 2024-03-29T11:18:56.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.929+0000 7f36352c1640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:56.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.929+0000 7f36352c1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:18:56.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.929+0000 7f36352c1640 10 monclient(hunting): picked mon.noname-a con 0x7f36300ae3c0 addr v1:172.21.15.17:6789/0 2024-03-29T11:18:56.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.929+0000 7f36352c1640 10 monclient(hunting): picked mon.noname-b con 0x7f36300aae90 addr v1:172.21.15.17:6790/0 2024-03-29T11:18:56.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.929+0000 7f36352c1640 10 monclient(hunting): picked mon.noname-c con 0x7f3630071dc0 addr v1:172.21.15.94:6789/0 2024-03-29T11:18:56.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.929+0000 7f36352c1640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36300ac7f0 con 0x7f36300ae3c0 2024-03-29T11:18:56.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.929+0000 7f36352c1640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36300ab370 con 0x7f3630071dc0 2024-03-29T11:18:56.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.929+0000 7f36352c1640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36300ab890 con 0x7f36300aae90 2024-03-29T11:18:56.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.929+0000 7f36352c1640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:56.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.929+0000 7f36352c1640 10 monclient(hunting): authenticate will time out at 822.090618s 2024-03-29T11:18:56.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.929+0000 7f362f7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f3630071dc0 0x7f363014a620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:38658/0 (socket says 172.21.15.94:38658) 2024-03-29T11:18:56.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.929+0000 7f362f7fe640 1 -- 172.21.15.94:0/1866236785 learned_addr learned my addr 172.21.15.94:0/1866236785 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T11:18:56.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.929+0000 7f362dffb640 1 -- 172.21.15.94:0/1866236785 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3939647407 0 0) 0x7f36300ab370 con 0x7f3630071dc0 2024-03-29T11:18:56.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.929+0000 7f362dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:18:56.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.929+0000 7f362dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:56.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.929+0000 7f362dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:56.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.929+0000 7f362dffb640 10 monclient(hunting): my global_id is 4117 2024-03-29T11:18:56.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.929+0000 7f362dffb640 1 -- 172.21.15.94:0/1866236785 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3618003520 con 0x7f3630071dc0 2024-03-29T11:18:56.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.930+0000 7f362dffb640 1 -- 172.21.15.94:0/1866236785 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2605952900 0 0) 0x7f36300ab890 con 0x7f36300aae90 2024-03-29T11:18:56.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.930+0000 7f362dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:18:56.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.930+0000 7f362dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:56.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.930+0000 7f362dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:56.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.930+0000 7f362dffb640 10 monclient(hunting): my global_id is 4115 2024-03-29T11:18:56.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.930+0000 7f362dffb640 1 -- 172.21.15.94:0/1866236785 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36300ab370 con 0x7f36300aae90 2024-03-29T11:18:56.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.930+0000 7f362dffb640 1 -- 172.21.15.94:0/1866236785 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1613656270 0 0) 0x7f36300ac7f0 con 0x7f36300ae3c0 2024-03-29T11:18:56.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.930+0000 7f362dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:18:56.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.930+0000 7f362dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:56.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.930+0000 7f362dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:56.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.930+0000 7f362dffb640 10 monclient(hunting): my global_id is 4113 2024-03-29T11:18:56.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.930+0000 7f362dffb640 1 -- 172.21.15.94:0/1866236785 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36300ab890 con 0x7f36300ae3c0 2024-03-29T11:18:56.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.930+0000 7f362dffb640 1 -- 172.21.15.94:0/1866236785 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2043088339 0 0) 0x7f3618003520 con 0x7f3630071dc0 2024-03-29T11:18:56.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.930+0000 7f362dffb640 1 -- 172.21.15.94:0/1866236785 >> v1:172.21.15.17:6790/0 conn(0x7f36300aae90 legacy=0x7f36300acef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:56.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.930+0000 7f362dffb640 1 -- 172.21.15.94:0/1866236785 >> v1:172.21.15.17:6789/0 conn(0x7f36300ae3c0 legacy=0x7f363013c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:56.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.930+0000 7f362dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:18:56.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.930+0000 7f362dffb640 1 monclient: found mon.noname-c 2024-03-29T11:18:56.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.930+0000 7f362dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:56.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.930+0000 7f362dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6789/0 2024-03-29T11:18:56.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.930+0000 7f362dffb640 1 -- 172.21.15.94:0/1866236785 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36300ad600 con 0x7f3630071dc0 2024-03-29T11:18:56.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.930+0000 7f362dffb640 10 monclient: _finish_auth 0 2024-03-29T11:18:56.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.930+0000 7f362dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:18:56.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.930+0000 7f362dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:56.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.930+0000 7f362dffb640 1 -- 172.21.15.94:0/1866236785 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f3628001a10 con 0x7f3630071dc0 2024-03-29T11:18:56.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.930+0000 7f36352c1640 5 monclient: authenticate success, global_id 4117 2024-03-29T11:18:56.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.930+0000 7f36352c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:56.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.930+0000 7f362dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:56.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.930+0000 7f362dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:18:56.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.930+0000 7f362dffb640 10 monclient: dump: 2024-03-29T11:18:56.918 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T11:18:56.918 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:56.918 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:56.918 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:56.918 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:56.918 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:56.918 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:56.918 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:56.918 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:56.919 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:56.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.930+0000 7f36352c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:56.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.930+0000 7f362dffb640 1 -- 172.21.15.94:0/1866236785 <== mon.1 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3628001bf0 con 0x7f3630071dc0 2024-03-29T11:18:56.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.930+0000 7f362dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:56.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.930+0000 7f362dffb640 1 -- 172.21.15.94:0/1866236785 <== mon.1 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f3628002000 con 0x7f3630071dc0 2024-03-29T11:18:56.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.930+0000 7f36352c1640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:56.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.930+0000 7f36352c1640 4 set_mon_vals no callback set 2024-03-29T11:18:56.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.930+0000 7f362dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:56.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.930+0000 7f362dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:18:56.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.930+0000 7f362dffb640 10 monclient: dump: 2024-03-29T11:18:56.919 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T11:18:56.919 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:56.919 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:56.919 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:56.920 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:56.920 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:56.920 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:56.920 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:56.920 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:56.920 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:56.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.930+0000 7f36352c1640 10 monclient: shutdown 2024-03-29T11:18:56.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.930+0000 7f36352c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:56.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.930+0000 7f36352c1640 1 -- 172.21.15.94:0/1866236785 >> v1:172.21.15.94:6789/0 conn(0x7f3630071dc0 legacy=0x7f363014a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:56.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.930+0000 7f36352c1640 1 -- 172.21.15.94:0/1866236785 shutdown_connections 2024-03-29T11:18:56.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.930+0000 7f36352c1640 1 -- 172.21.15.94:0/1866236785 >> 172.21.15.94:0/1866236785 conn(0x7f36300b3750 msgr2=0x7f36300b3b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:56.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.930+0000 7f36352c1640 1 -- 172.21.15.94:0/1866236785 shutdown_connections 2024-03-29T11:18:56.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.930+0000 7f36352c1640 1 -- 172.21.15.94:0/1866236785 wait complete. 2024-03-29T11:18:56.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.931+0000 7f36352c1640 10 monclient: build_initial_monmap 2024-03-29T11:18:56.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.931+0000 7f36352c1640 10 monclient: monmap: 2024-03-29T11:18:56.921 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-03-29T11:18:56.921 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:56.921 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-03-29T11:18:56.921 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-03-29T11:18:56.921 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:56.921 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:56.921 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:18:56.921 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:18:56.921 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:18:56.921 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:56.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.931+0000 7f36352c1640 1 librados: starting msgr at 2024-03-29T11:18:56.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.931+0000 7f36352c1640 1 librados: starting objecter 2024-03-29T11:18:56.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.932+0000 7f36352c1640 1 Processor -- start 2024-03-29T11:18:56.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.932+0000 7f36352c1640 1 -- start start 2024-03-29T11:18:56.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.932+0000 7f36352c1640 1 librados: setting wanted keys 2024-03-29T11:18:56.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.932+0000 7f36352c1640 1 librados: calling monclient init 2024-03-29T11:18:56.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.932+0000 7f36352c1640 10 monclient: init 2024-03-29T11:18:56.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.932+0000 7f36352c1640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:56.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.932+0000 7f36352c1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:18:56.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.932+0000 7f36352c1640 10 monclient(hunting): picked mon.noname-b con 0x7f3630071dc0 addr v1:172.21.15.94:6789/0 2024-03-29T11:18:56.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.932+0000 7f36352c1640 10 monclient(hunting): picked mon.noname-c con 0x7f36300aae90 addr v1:172.21.15.17:6790/0 2024-03-29T11:18:56.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.932+0000 7f36352c1640 10 monclient(hunting): picked mon.noname-a con 0x7f36300ae3c0 addr v1:172.21.15.17:6789/0 2024-03-29T11:18:56.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.932+0000 7f36352c1640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36300ada00 con 0x7f36300ae3c0 2024-03-29T11:18:56.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.932+0000 7f36352c1640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36301d6c90 con 0x7f3630071dc0 2024-03-29T11:18:56.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.932+0000 7f36352c1640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3630070100 con 0x7f36300aae90 2024-03-29T11:18:56.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.932+0000 7f36352c1640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:56.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.932+0000 7f36352c1640 10 monclient(hunting): authenticate will time out at 822.093621s 2024-03-29T11:18:56.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.932+0000 7f362effd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f3630071dc0 0x7f36301d7120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:38662/0 (socket says 172.21.15.94:38662) 2024-03-29T11:18:56.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.932+0000 7f362effd640 1 -- 172.21.15.94:0/1478576907 learned_addr learned my addr 172.21.15.94:0/1478576907 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T11:18:56.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.932+0000 7f36177fe640 10 client.?.objecter ms_handle_connect 0x7f3630071dc0 2024-03-29T11:18:56.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.932+0000 7f36177fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:56.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.932+0000 7f36177fe640 10 client.?.objecter ms_handle_connect 0x7f36300ae3c0 2024-03-29T11:18:56.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.932+0000 7f36177fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:56.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.932+0000 7f36177fe640 10 client.?.objecter ms_handle_connect 0x7f36300aae90 2024-03-29T11:18:56.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.932+0000 7f36177fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:56.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.932+0000 7f36177fe640 1 -- 172.21.15.94:0/1478576907 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3475824072 0 0) 0x7f36301d6c90 con 0x7f3630071dc0 2024-03-29T11:18:56.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.932+0000 7f36177fe640 10 monclient(hunting): _negotiate 2024-03-29T11:18:56.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.932+0000 7f36177fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:56.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.932+0000 7f36177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:56.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.932+0000 7f36177fe640 10 monclient(hunting): my global_id is 4120 2024-03-29T11:18:56.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.932+0000 7f36177fe640 1 -- 172.21.15.94:0/1478576907 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f361c004eb0 con 0x7f3630071dc0 2024-03-29T11:18:56.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.932+0000 7f36177fe640 1 -- 172.21.15.94:0/1478576907 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2662733578 0 0) 0x7f3630070100 con 0x7f36300aae90 2024-03-29T11:18:56.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.932+0000 7f36177fe640 10 monclient(hunting): _negotiate 2024-03-29T11:18:56.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.932+0000 7f36177fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:56.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.932+0000 7f36177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:56.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.932+0000 7f36177fe640 10 monclient(hunting): my global_id is 4118 2024-03-29T11:18:56.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.932+0000 7f36177fe640 1 -- 172.21.15.94:0/1478576907 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36301d6c90 con 0x7f36300aae90 2024-03-29T11:18:56.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36177fe640 1 -- 172.21.15.94:0/1478576907 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 942405635 0 0) 0x7f36300ada00 con 0x7f36300ae3c0 2024-03-29T11:18:56.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36177fe640 10 monclient(hunting): _negotiate 2024-03-29T11:18:56.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36177fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:56.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:56.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36177fe640 10 monclient(hunting): my global_id is 4116 2024-03-29T11:18:56.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36177fe640 1 -- 172.21.15.94:0/1478576907 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3630070100 con 0x7f36300ae3c0 2024-03-29T11:18:56.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36177fe640 1 -- 172.21.15.94:0/1478576907 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 352235579 0 0) 0x7f361c004eb0 con 0x7f3630071dc0 2024-03-29T11:18:56.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36177fe640 1 -- 172.21.15.94:0/1478576907 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36300ada00 con 0x7f3630071dc0 2024-03-29T11:18:56.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36177fe640 1 -- 172.21.15.94:0/1478576907 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f3624001970 con 0x7f3630071dc0 2024-03-29T11:18:56.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:56.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36177fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:18:56.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36177fe640 10 monclient(hunting): dump: 2024-03-29T11:18:56.925 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T11:18:56.925 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:56.925 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:56.925 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:56.925 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:56.925 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:56.925 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:56.925 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:56.925 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:56.925 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:56.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36177fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:56.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36177fe640 1 -- 172.21.15.94:0/1478576907 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3030363122 0 0) 0x7f36301d6c90 con 0x7f36300aae90 2024-03-29T11:18:56.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36177fe640 1 -- 172.21.15.94:0/1478576907 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f361c004eb0 con 0x7f36300aae90 2024-03-29T11:18:56.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36177fe640 1 -- 172.21.15.94:0/1478576907 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f3620001930 con 0x7f36300aae90 2024-03-29T11:18:56.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:56.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36177fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:56.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36177fe640 10 monclient(hunting): dump: 2024-03-29T11:18:56.926 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T11:18:56.926 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:56.926 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:56.926 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:56.926 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:56.926 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:56.926 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:56.926 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:56.927 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:56.927 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:56.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36177fe640 1 -- 172.21.15.94:0/1478576907 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3136941860 0 0) 0x7f3630070100 con 0x7f36300ae3c0 2024-03-29T11:18:56.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36177fe640 1 -- 172.21.15.94:0/1478576907 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36301d6c90 con 0x7f36300ae3c0 2024-03-29T11:18:56.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36177fe640 1 -- 172.21.15.94:0/1478576907 <== mon.1 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2870589114 0 0) 0x7f36300ada00 con 0x7f3630071dc0 2024-03-29T11:18:56.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36177fe640 1 -- 172.21.15.94:0/1478576907 >> v1:172.21.15.17:6790/0 conn(0x7f36300aae90 legacy=0x7f3630070ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:56.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36177fe640 1 -- 172.21.15.94:0/1478576907 >> v1:172.21.15.17:6789/0 conn(0x7f36300ae3c0 legacy=0x7f363006e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:56.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36177fe640 10 monclient: _finish_hunting 0 2024-03-29T11:18:56.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36177fe640 1 monclient: found mon.b 2024-03-29T11:18:56.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:56.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36177fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:18:56.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36177fe640 1 -- 172.21.15.94:0/1478576907 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36301d7830 con 0x7f3630071dc0 2024-03-29T11:18:56.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36177fe640 10 monclient: _finish_auth 0 2024-03-29T11:18:56.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36177fe640 10 monclient: _check_auth_tickets 2024-03-29T11:18:56.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:56.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36352c1640 5 monclient: authenticate success, global_id 4120 2024-03-29T11:18:56.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36352c1640 10 monclient: _renew_subs 2024-03-29T11:18:56.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36352c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:18:56.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36352c1640 1 -- 172.21.15.94:0/1478576907 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36300713b0 con 0x7f3630071dc0 2024-03-29T11:18:56.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36352c1640 10 client.4120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:56.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36352c1640 10 monclient: _renew_subs 2024-03-29T11:18:56.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36352c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:18:56.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36352c1640 1 -- 172.21.15.94:0/1478576907 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3630070700 con 0x7f3630071dc0 2024-03-29T11:18:56.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36352c1640 1 librados: init done 2024-03-29T11:18:56.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36177fe640 1 -- 172.21.15.94:0/1478576907 <== mon.1 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3624001c50 con 0x7f3630071dc0 2024-03-29T11:18:56.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36177fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:56.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f362d7fa640 4 set_mon_vals no callback set 2024-03-29T11:18:56.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36177fe640 1 -- 172.21.15.94:0/1478576907 <== mon.1 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f3624003a60 con 0x7f3630071dc0 2024-03-29T11:18:56.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:56.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36177fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:18:56.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36177fe640 10 monclient: dump: 2024-03-29T11:18:56.929 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T11:18:56.929 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:56.929 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:56.929 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:56.929 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:56.929 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:56.929 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:56.929 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:56.929 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:56.929 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:56.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36177fe640 1 -- 172.21.15.94:0/1478576907 <== mon.1 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f3624004050 con 0x7f3630071dc0 2024-03-29T11:18:56.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36177fe640 1 -- 172.21.15.94:0/1478576907 <== mon.1 v1:172.21.15.94:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 2038227367 0 0) 0x7f3624006590 con 0x7f3630071dc0 2024-03-29T11:18:56.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36177fe640 10 client.4120.objecter ms_dispatch 0x7f3630000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T11:18:56.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36177fe640 3 client.4120.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T11:18:56.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36177fe640 3 client.4120.objecter handle_osd_map decoding full epoch 2 2024-03-29T11:18:56.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.933+0000 7f36177fe640 20 client.4120.objecter dump_active .. 0 homeless 2024-03-29T11:18:56.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.934+0000 7f36352c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:56.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.934+0000 7f36352c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:56.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.934+0000 7f36352c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:18:56.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.934+0000 7f36352c1640 1 -- 172.21.15.94:0/1478576907 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3610005080 con 0x7f3630071dc0 2024-03-29T11:18:56.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.936+0000 7f36177fe640 1 -- 172.21.15.94:0/1478576907 <== mon.1 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f36240057c0 con 0x7f3630071dc0 2024-03-29T11:18:56.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.936+0000 7f36177fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:56.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:56.936+0000 7f36177fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:56.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.008+0000 7f36352c1640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e7583f99-c8fc-47b7-98c9-000b5892563e", "id": 0}] 2024-03-29T11:18:56.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.008+0000 7f36352c1640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e7583f99-c8fc-47b7-98c9-000b5892563e", "id": 0}] 2024-03-29T11:18:56.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.008+0000 7f36352c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:18:56.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.008+0000 7f36352c1640 1 -- 172.21.15.94:0/1478576907 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd new", "uuid": "e7583f99-c8fc-47b7-98c9-000b5892563e", "id": 0} v 0) v1 -- 0x7f36100058b0 con 0x7f3630071dc0 2024-03-29T11:18:56.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.011+0000 7f36177fe640 1 -- 172.21.15.94:0/1478576907 <== mon.1 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e7583f99-c8fc-47b7-98c9-000b5892563e", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 2095688231 0 732707515) 0x7f36240059e0 con 0x7f3630071dc0 2024-03-29T11:18:56.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.011+0000 7f36177fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e7583f99-c8fc-47b7-98c9-000b5892563e", "id": 0}] 2024-03-29T11:18:56.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.011+0000 7f36177fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:56.988 INFO:teuthology.orchestra.run.smithi094.stdout:0 2024-03-29T11:18:56.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.011+0000 7f36352c1640 10 librados: watch_flush enter 2024-03-29T11:18:56.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.012+0000 7f36352c1640 10 librados: watch_flush exit 2024-03-29T11:18:56.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.012+0000 7f36352c1640 20 client.4120.objecter shutdown clearing up homeless session... 2024-03-29T11:18:56.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.012+0000 7f36352c1640 10 client.4120.objecter successfully canceled tick 2024-03-29T11:18:56.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.012+0000 7f36352c1640 10 monclient: shutdown 2024-03-29T11:18:56.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.012+0000 7f36352c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:56.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.012+0000 7f36352c1640 1 -- 172.21.15.94:0/1478576907 >> v1:172.21.15.94:6789/0 conn(0x7f3630071dc0 legacy=0x7f36301d7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:56.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.012+0000 7f36352c1640 1 -- 172.21.15.94:0/1478576907 shutdown_connections 2024-03-29T11:18:56.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.012+0000 7f36352c1640 1 -- 172.21.15.94:0/1478576907 >> 172.21.15.94:0/1478576907 conn(0x7f36300b3750 msgr2=0x7f36300a9cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:56.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.012+0000 7f36352c1640 1 -- 172.21.15.94:0/1478576907 shutdown_connections 2024-03-29T11:18:56.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.012+0000 7f36352c1640 1 -- 172.21.15.94:0/1478576907 wait complete. 2024-03-29T11:18:56.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.012+0000 7f36352c1640 1 librados: shutdown 2024-03-29T11:18:56.999 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster ceph osd new 2b167bf1-6458-4b07-b603-ecceaf19933e 1 2024-03-29T11:18:57.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.122+0000 7f03a034b640 10 monclient: get_monmap_and_config 2024-03-29T11:18:57.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.122+0000 7f03a034b640 10 monclient: build_initial_monmap 2024-03-29T11:18:57.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.122+0000 7f03a034b640 10 monclient: monmap: 2024-03-29T11:18:57.099 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-03-29T11:18:57.099 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:57.099 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:57.123014+0000 2024-03-29T11:18:57.099 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:57.123014+0000 2024-03-29T11:18:57.099 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:57.099 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:57.099 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:18:57.099 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:18:57.099 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:18:57.099 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:57.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.122+0000 7f03a034b640 1 Processor -- start 2024-03-29T11:18:57.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.122+0000 7f03a034b640 1 -- start start 2024-03-29T11:18:57.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.122+0000 7f03a034b640 10 monclient: init 2024-03-29T11:18:57.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.123+0000 7f03a034b640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:57.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.123+0000 7f03a034b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:18:57.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.123+0000 7f03a034b640 10 monclient(hunting): picked mon.noname-c con 0x7f039809ac40 addr v1:172.21.15.94:6789/0 2024-03-29T11:18:57.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.123+0000 7f03a034b640 10 monclient(hunting): picked mon.noname-a con 0x7f039809c870 addr v1:172.21.15.17:6789/0 2024-03-29T11:18:57.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.123+0000 7f03a034b640 10 monclient(hunting): picked mon.noname-b con 0x7f03980b4360 addr v1:172.21.15.17:6790/0 2024-03-29T11:18:57.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.123+0000 7f03a034b640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f039809dec0 con 0x7f039809c870 2024-03-29T11:18:57.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.123+0000 7f03a034b640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f039809da30 con 0x7f039809ac40 2024-03-29T11:18:57.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.123+0000 7f03a034b640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03980b39c0 con 0x7f03980b4360 2024-03-29T11:18:57.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.123+0000 7f03a034b640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:57.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.123+0000 7f03a034b640 10 monclient(hunting): authenticate will time out at 822.284508s 2024-03-29T11:18:57.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.123+0000 7f039e0e7640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f039809ac40 0x7f03980b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:38678/0 (socket says 172.21.15.94:38678) 2024-03-29T11:18:57.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.123+0000 7f039e0e7640 1 -- 172.21.15.94:0/3742849544 learned_addr learned my addr 172.21.15.94:0/3742849544 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T11:18:57.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.123+0000 7f039d0e5640 1 -- 172.21.15.94:0/3742849544 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2274183767 0 0) 0x7f039809da30 con 0x7f039809ac40 2024-03-29T11:18:57.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.123+0000 7f039d0e5640 10 monclient(hunting): _negotiate 2024-03-29T11:18:57.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.123+0000 7f039d0e5640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:57.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.123+0000 7f039d0e5640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:57.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.123+0000 7f039d0e5640 10 monclient(hunting): my global_id is 4123 2024-03-29T11:18:57.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.123+0000 7f039d0e5640 1 -- 172.21.15.94:0/3742849544 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0384003520 con 0x7f039809ac40 2024-03-29T11:18:57.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.123+0000 7f039d0e5640 1 -- 172.21.15.94:0/3742849544 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1275691917 0 0) 0x7f0394001110 con 0x7f039809c870 2024-03-29T11:18:57.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.123+0000 7f039d0e5640 10 monclient(hunting): _negotiate 2024-03-29T11:18:57.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.123+0000 7f039d0e5640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:57.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.123+0000 7f039d0e5640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:57.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.123+0000 7f039d0e5640 10 monclient(hunting): my global_id is 4119 2024-03-29T11:18:57.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.123+0000 7f039d0e5640 1 -- 172.21.15.94:0/3742849544 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f039809da30 con 0x7f039809c870 2024-03-29T11:18:57.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.123+0000 7f039d0e5640 1 -- 172.21.15.94:0/3742849544 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2105086548 0 0) 0x7f03980b39c0 con 0x7f03980b4360 2024-03-29T11:18:57.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.123+0000 7f039d0e5640 10 monclient(hunting): _negotiate 2024-03-29T11:18:57.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.123+0000 7f039d0e5640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:57.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.123+0000 7f039d0e5640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:57.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.123+0000 7f039d0e5640 10 monclient(hunting): my global_id is 4121 2024-03-29T11:18:57.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.123+0000 7f039d0e5640 1 -- 172.21.15.94:0/3742849544 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0394001110 con 0x7f03980b4360 2024-03-29T11:18:57.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.124+0000 7f039d0e5640 1 -- 172.21.15.94:0/3742849544 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2128042279 0 0) 0x7f0384003520 con 0x7f039809ac40 2024-03-29T11:18:57.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.124+0000 7f039d0e5640 1 -- 172.21.15.94:0/3742849544 >> v1:172.21.15.17:6790/0 conn(0x7f03980b4360 legacy=0x7f039814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:57.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.124+0000 7f039d0e5640 1 -- 172.21.15.94:0/3742849544 >> v1:172.21.15.17:6789/0 conn(0x7f039809c870 legacy=0x7f039813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:57.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.124+0000 7f039d0e5640 10 monclient: _finish_hunting 0 2024-03-29T11:18:57.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.124+0000 7f039d0e5640 1 monclient: found mon.noname-c 2024-03-29T11:18:57.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.124+0000 7f039d0e5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:57.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.124+0000 7f039d0e5640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6789/0 2024-03-29T11:18:57.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.124+0000 7f039d0e5640 1 -- 172.21.15.94:0/3742849544 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03980b5160 con 0x7f039809ac40 2024-03-29T11:18:57.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.124+0000 7f039d0e5640 10 monclient: _finish_auth 0 2024-03-29T11:18:57.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.124+0000 7f039d0e5640 10 monclient: _check_auth_tickets 2024-03-29T11:18:57.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.124+0000 7f039d0e5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:57.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.124+0000 7f03a034b640 5 monclient: authenticate success, global_id 4123 2024-03-29T11:18:57.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.124+0000 7f039d0e5640 1 -- 172.21.15.94:0/3742849544 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f038c001690 con 0x7f039809ac40 2024-03-29T11:18:57.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.124+0000 7f03a034b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:57.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.124+0000 7f039d0e5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:57.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.124+0000 7f039d0e5640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:18:57.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.124+0000 7f039d0e5640 10 monclient: dump: 2024-03-29T11:18:57.111 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T11:18:57.111 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:57.111 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.111 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.111 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:57.111 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:57.111 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:57.112 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:57.112 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:57.112 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:57.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.124+0000 7f03a034b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:57.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.124+0000 7f039d0e5640 1 -- 172.21.15.94:0/3742849544 <== mon.1 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f038c001870 con 0x7f039809ac40 2024-03-29T11:18:57.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.124+0000 7f039d0e5640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:57.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.124+0000 7f039d0e5640 1 -- 172.21.15.94:0/3742849544 <== mon.1 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f038c001c80 con 0x7f039809ac40 2024-03-29T11:18:57.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.124+0000 7f03a034b640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:57.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.124+0000 7f03a034b640 4 set_mon_vals no callback set 2024-03-29T11:18:57.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.124+0000 7f039d0e5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:57.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.124+0000 7f039d0e5640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:18:57.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.124+0000 7f039d0e5640 10 monclient: dump: 2024-03-29T11:18:57.112 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T11:18:57.112 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:57.113 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.113 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.113 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:57.113 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:57.113 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:57.113 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:57.113 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:57.113 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:57.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.124+0000 7f03a034b640 10 monclient: shutdown 2024-03-29T11:18:57.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.124+0000 7f03a034b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:57.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.124+0000 7f03a034b640 1 -- 172.21.15.94:0/3742849544 >> v1:172.21.15.94:6789/0 conn(0x7f039809ac40 legacy=0x7f03980b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:57.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.124+0000 7f03a034b640 1 -- 172.21.15.94:0/3742849544 shutdown_connections 2024-03-29T11:18:57.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.124+0000 7f03a034b640 1 -- 172.21.15.94:0/3742849544 >> 172.21.15.94:0/3742849544 conn(0x7f0398066b00 msgr2=0x7f0398066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:57.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.124+0000 7f03a034b640 1 -- 172.21.15.94:0/3742849544 shutdown_connections 2024-03-29T11:18:57.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.124+0000 7f03a034b640 1 -- 172.21.15.94:0/3742849544 wait complete. 2024-03-29T11:18:57.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.125+0000 7f03a034b640 10 monclient: build_initial_monmap 2024-03-29T11:18:57.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.125+0000 7f03a034b640 10 monclient: monmap: 2024-03-29T11:18:57.114 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-03-29T11:18:57.114 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:57.114 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-03-29T11:18:57.114 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-03-29T11:18:57.114 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:57.114 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:57.114 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:18:57.114 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:18:57.114 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:18:57.114 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:57.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.125+0000 7f03a034b640 1 librados: starting msgr at 2024-03-29T11:18:57.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.125+0000 7f03a034b640 1 librados: starting objecter 2024-03-29T11:18:57.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.125+0000 7f03a034b640 1 Processor -- start 2024-03-29T11:18:57.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.125+0000 7f03a034b640 1 -- start start 2024-03-29T11:18:57.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.125+0000 7f03a034b640 1 librados: setting wanted keys 2024-03-29T11:18:57.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.125+0000 7f03a034b640 1 librados: calling monclient init 2024-03-29T11:18:57.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.125+0000 7f03a034b640 10 monclient: init 2024-03-29T11:18:57.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.126+0000 7f03a034b640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:57.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.126+0000 7f03a034b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:18:57.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.126+0000 7f03a034b640 10 monclient(hunting): picked mon.noname-c con 0x7f039809ac40 addr v1:172.21.15.17:6790/0 2024-03-29T11:18:57.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.126+0000 7f03a034b640 10 monclient(hunting): picked mon.noname-b con 0x7f03980b1280 addr v1:172.21.15.94:6789/0 2024-03-29T11:18:57.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.126+0000 7f03a034b640 10 monclient(hunting): picked mon.noname-a con 0x7f03981d6ed0 addr v1:172.21.15.17:6789/0 2024-03-29T11:18:57.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.126+0000 7f03a034b640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03981422a0 con 0x7f03981d6ed0 2024-03-29T11:18:57.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.126+0000 7f03a034b640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0398142f50 con 0x7f03980b1280 2024-03-29T11:18:57.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.126+0000 7f03a034b640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03981ccab0 con 0x7f039809ac40 2024-03-29T11:18:57.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.126+0000 7f03a034b640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:57.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.126+0000 7f03a034b640 10 monclient(hunting): authenticate will time out at 822.287417s 2024-03-29T11:18:57.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.126+0000 7f039d8e6640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f03980b1280 0x7f0398142840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:38682/0 (socket says 172.21.15.94:38682) 2024-03-29T11:18:57.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.126+0000 7f039d8e6640 1 -- 172.21.15.94:0/713857796 learned_addr learned my addr 172.21.15.94:0/713857796 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T11:18:57.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.126+0000 7f038affd640 10 client.?.objecter ms_handle_connect 0x7f03980b1280 2024-03-29T11:18:57.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.126+0000 7f038affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:57.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.126+0000 7f038affd640 10 client.?.objecter ms_handle_connect 0x7f039809ac40 2024-03-29T11:18:57.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.126+0000 7f038affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:57.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.126+0000 7f038affd640 10 client.?.objecter ms_handle_connect 0x7f03981d6ed0 2024-03-29T11:18:57.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.126+0000 7f038affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:57.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.126+0000 7f038affd640 1 -- 172.21.15.94:0/713857796 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2806131332 0 0) 0x7f0398142f50 con 0x7f03980b1280 2024-03-29T11:18:57.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.126+0000 7f038affd640 10 monclient(hunting): _negotiate 2024-03-29T11:18:57.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.126+0000 7f038affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:57.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.126+0000 7f038affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:57.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.126+0000 7f038affd640 10 monclient(hunting): my global_id is 4126 2024-03-29T11:18:57.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.126+0000 7f038affd640 1 -- 172.21.15.94:0/713857796 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f037c004eb0 con 0x7f03980b1280 2024-03-29T11:18:57.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.126+0000 7f038affd640 1 -- 172.21.15.94:0/713857796 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2473660406 0 0) 0x7f03981ccab0 con 0x7f039809ac40 2024-03-29T11:18:57.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.126+0000 7f038affd640 10 monclient(hunting): _negotiate 2024-03-29T11:18:57.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.126+0000 7f038affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:57.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.126+0000 7f038affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:57.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.126+0000 7f038affd640 10 monclient(hunting): my global_id is 4124 2024-03-29T11:18:57.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.126+0000 7f038affd640 1 -- 172.21.15.94:0/713857796 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0398142f50 con 0x7f039809ac40 2024-03-29T11:18:57.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.126+0000 7f038affd640 1 -- 172.21.15.94:0/713857796 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1616066003 0 0) 0x7f03981422a0 con 0x7f03981d6ed0 2024-03-29T11:18:57.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.126+0000 7f038affd640 10 monclient(hunting): _negotiate 2024-03-29T11:18:57.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.126+0000 7f038affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:57.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.126+0000 7f038affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:57.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.126+0000 7f038affd640 10 monclient(hunting): my global_id is 4122 2024-03-29T11:18:57.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.126+0000 7f038affd640 1 -- 172.21.15.94:0/713857796 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03981ccab0 con 0x7f03981d6ed0 2024-03-29T11:18:57.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.126+0000 7f038affd640 1 -- 172.21.15.94:0/713857796 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3132611405 0 0) 0x7f037c004eb0 con 0x7f03980b1280 2024-03-29T11:18:57.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.126+0000 7f038affd640 1 -- 172.21.15.94:0/713857796 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f03981422a0 con 0x7f03980b1280 2024-03-29T11:18:57.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.126+0000 7f038affd640 1 -- 172.21.15.94:0/713857796 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f03940019a0 con 0x7f03980b1280 2024-03-29T11:18:57.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.126+0000 7f038affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:57.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.126+0000 7f038affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:18:57.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.126+0000 7f038affd640 10 monclient(hunting): dump: 2024-03-29T11:18:57.118 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T11:18:57.118 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:57.118 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.118 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.119 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:57.119 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:57.119 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:57.119 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:57.119 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:57.119 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:57.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.126+0000 7f038affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:57.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.126+0000 7f038affd640 1 -- 172.21.15.94:0/713857796 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4219647466 0 0) 0x7f0398142f50 con 0x7f039809ac40 2024-03-29T11:18:57.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.127+0000 7f038affd640 1 -- 172.21.15.94:0/713857796 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f037c004eb0 con 0x7f039809ac40 2024-03-29T11:18:57.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.127+0000 7f038affd640 1 -- 172.21.15.94:0/713857796 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f038c001b80 con 0x7f039809ac40 2024-03-29T11:18:57.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.127+0000 7f038affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:57.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.127+0000 7f038affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:57.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.127+0000 7f038affd640 10 monclient(hunting): dump: 2024-03-29T11:18:57.119 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T11:18:57.120 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:57.120 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.120 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.120 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:57.120 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:57.120 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:57.120 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:57.120 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:57.120 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:57.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.127+0000 7f038affd640 1 -- 172.21.15.94:0/713857796 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2922730665 0 0) 0x7f03981ccab0 con 0x7f03981d6ed0 2024-03-29T11:18:57.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.127+0000 7f038affd640 1 -- 172.21.15.94:0/713857796 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0398142f50 con 0x7f03981d6ed0 2024-03-29T11:18:57.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.127+0000 7f038affd640 1 -- 172.21.15.94:0/713857796 <== mon.1 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1041534660 0 0) 0x7f03981422a0 con 0x7f03980b1280 2024-03-29T11:18:57.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.127+0000 7f038affd640 1 -- 172.21.15.94:0/713857796 >> v1:172.21.15.17:6790/0 conn(0x7f039809ac40 legacy=0x7f0398141ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:57.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.127+0000 7f038affd640 1 -- 172.21.15.94:0/713857796 >> v1:172.21.15.17:6789/0 conn(0x7f03981d6ed0 legacy=0x7f039813adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:57.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.127+0000 7f038affd640 10 monclient: _finish_hunting 0 2024-03-29T11:18:57.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.127+0000 7f038affd640 1 monclient: found mon.b 2024-03-29T11:18:57.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.127+0000 7f038affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:57.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.127+0000 7f038affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:18:57.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.127+0000 7f038affd640 1 -- 172.21.15.94:0/713857796 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03981ccc40 con 0x7f03980b1280 2024-03-29T11:18:57.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.127+0000 7f038affd640 10 monclient: _finish_auth 0 2024-03-29T11:18:57.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.127+0000 7f038affd640 10 monclient: _check_auth_tickets 2024-03-29T11:18:57.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.127+0000 7f038affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:57.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.127+0000 7f03a034b640 5 monclient: authenticate success, global_id 4126 2024-03-29T11:18:57.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.127+0000 7f03a034b640 10 monclient: _renew_subs 2024-03-29T11:18:57.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.127+0000 7f03a034b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:18:57.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.127+0000 7f03a034b640 1 -- 172.21.15.94:0/713857796 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f039813b4d0 con 0x7f03980b1280 2024-03-29T11:18:57.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.127+0000 7f03a034b640 10 client.4126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:57.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.127+0000 7f03a034b640 10 monclient: _renew_subs 2024-03-29T11:18:57.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.127+0000 7f03a034b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:18:57.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.127+0000 7f03a034b640 1 -- 172.21.15.94:0/713857796 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f03981e3720 con 0x7f03980b1280 2024-03-29T11:18:57.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.127+0000 7f03a034b640 1 librados: init done 2024-03-29T11:18:57.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.127+0000 7f038affd640 1 -- 172.21.15.94:0/713857796 <== mon.1 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0394003610 con 0x7f03980b1280 2024-03-29T11:18:57.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.127+0000 7f038affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:57.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.127+0000 7f039c8e4640 4 set_mon_vals no callback set 2024-03-29T11:18:57.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.127+0000 7f038affd640 1 -- 172.21.15.94:0/713857796 <== mon.1 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f0394003920 con 0x7f03980b1280 2024-03-29T11:18:57.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.127+0000 7f038affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:57.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.127+0000 7f038affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:18:57.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.127+0000 7f038affd640 10 monclient: dump: 2024-03-29T11:18:57.122 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T11:18:57.122 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:57.122 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.122 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.122 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:57.123 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:57.123 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:57.123 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:57.123 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:57.123 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:57.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.127+0000 7f038affd640 1 -- 172.21.15.94:0/713857796 <== mon.1 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f0394003e10 con 0x7f03980b1280 2024-03-29T11:18:57.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.127+0000 7f038affd640 1 -- 172.21.15.94:0/713857796 <== mon.1 v1:172.21.15.94:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 2897529719 0 0) 0x7f0394006560 con 0x7f03980b1280 2024-03-29T11:18:57.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.127+0000 7f038affd640 10 client.4126.objecter ms_dispatch 0x7f0398000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T11:18:57.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.127+0000 7f038affd640 3 client.4126.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T11:18:57.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.127+0000 7f038affd640 3 client.4126.objecter handle_osd_map decoding full epoch 3 2024-03-29T11:18:57.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.127+0000 7f038affd640 20 client.4126.objecter dump_active .. 0 homeless 2024-03-29T11:18:57.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.128+0000 7f03a034b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:57.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.128+0000 7f03a034b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:57.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.128+0000 7f03a034b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:18:57.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.128+0000 7f03a034b640 1 -- 172.21.15.94:0/713857796 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0380005080 con 0x7f03980b1280 2024-03-29T11:18:57.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.130+0000 7f038affd640 1 -- 172.21.15.94:0/713857796 <== mon.1 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0394003780 con 0x7f03980b1280 2024-03-29T11:18:57.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.130+0000 7f038affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:57.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.130+0000 7f038affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:57.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.201+0000 7f03a034b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "2b167bf1-6458-4b07-b603-ecceaf19933e", "id": 1}] 2024-03-29T11:18:57.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.201+0000 7f03a034b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "2b167bf1-6458-4b07-b603-ecceaf19933e", "id": 1}] 2024-03-29T11:18:57.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.201+0000 7f03a034b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:18:57.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.201+0000 7f03a034b640 1 -- 172.21.15.94:0/713857796 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd new", "uuid": "2b167bf1-6458-4b07-b603-ecceaf19933e", "id": 1} v 0) v1 -- 0x7f03800058b0 con 0x7f03980b1280 2024-03-29T11:18:57.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.204+0000 7f038affd640 1 -- 172.21.15.94:0/713857796 <== mon.1 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2b167bf1-6458-4b07-b603-ecceaf19933e", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 2657685221 0 940483276) 0x7f03940057c0 con 0x7f03980b1280 2024-03-29T11:18:57.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.204+0000 7f038affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "2b167bf1-6458-4b07-b603-ecceaf19933e", "id": 1}] 2024-03-29T11:18:57.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.204+0000 7f038affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:57.181 INFO:teuthology.orchestra.run.smithi094.stdout:1 2024-03-29T11:18:57.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.205+0000 7f03a034b640 10 librados: watch_flush enter 2024-03-29T11:18:57.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.205+0000 7f03a034b640 10 librados: watch_flush exit 2024-03-29T11:18:57.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.205+0000 7f03a034b640 20 client.4126.objecter shutdown clearing up homeless session... 2024-03-29T11:18:57.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.205+0000 7f03a034b640 10 client.4126.objecter successfully canceled tick 2024-03-29T11:18:57.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.205+0000 7f03a034b640 10 monclient: shutdown 2024-03-29T11:18:57.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.205+0000 7f03a034b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:57.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.205+0000 7f03a034b640 1 -- 172.21.15.94:0/713857796 >> v1:172.21.15.94:6789/0 conn(0x7f03980b1280 legacy=0x7f0398142840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:57.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.205+0000 7f03a034b640 1 -- 172.21.15.94:0/713857796 shutdown_connections 2024-03-29T11:18:57.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.205+0000 7f03a034b640 1 -- 172.21.15.94:0/713857796 >> 172.21.15.94:0/713857796 conn(0x7f0398066b00 msgr2=0x7f0398066ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:57.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.205+0000 7f03a034b640 1 -- 172.21.15.94:0/713857796 shutdown_connections 2024-03-29T11:18:57.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.205+0000 7f03a034b640 1 -- 172.21.15.94:0/713857796 wait complete. 2024-03-29T11:18:57.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.205+0000 7f03a034b640 1 librados: shutdown 2024-03-29T11:18:57.192 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster ceph osd new b2fd6920-a919-4184-bce2-1aee9c71f90e 2 2024-03-29T11:18:57.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.315+0000 7f7f1d3cd640 10 monclient: get_monmap_and_config 2024-03-29T11:18:57.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.315+0000 7f7f1d3cd640 10 monclient: build_initial_monmap 2024-03-29T11:18:57.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.315+0000 7f7f1d3cd640 10 monclient: monmap: 2024-03-29T11:18:57.292 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-03-29T11:18:57.292 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:57.292 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:57.316491+0000 2024-03-29T11:18:57.292 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:57.316491+0000 2024-03-29T11:18:57.293 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:57.293 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:57.293 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:18:57.293 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:18:57.293 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:18:57.293 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:57.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.316+0000 7f7f1d3cd640 1 Processor -- start 2024-03-29T11:18:57.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.316+0000 7f7f1d3cd640 1 -- start start 2024-03-29T11:18:57.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.316+0000 7f7f1d3cd640 10 monclient: init 2024-03-29T11:18:57.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.316+0000 7f7f1d3cd640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:57.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.316+0000 7f7f1d3cd640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:18:57.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.316+0000 7f7f1d3cd640 10 monclient(hunting): picked mon.noname-b con 0x7f7f1809ac40 addr v1:172.21.15.17:6790/0 2024-03-29T11:18:57.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.316+0000 7f7f1d3cd640 10 monclient(hunting): picked mon.noname-a con 0x7f7f1809c870 addr v1:172.21.15.17:6789/0 2024-03-29T11:18:57.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.316+0000 7f7f1d3cd640 10 monclient(hunting): picked mon.noname-c con 0x7f7f180b4360 addr v1:172.21.15.94:6789/0 2024-03-29T11:18:57.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.316+0000 7f7f1d3cd640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f1809dec0 con 0x7f7f1809c870 2024-03-29T11:18:57.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.316+0000 7f7f1d3cd640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f1809da30 con 0x7f7f180b4360 2024-03-29T11:18:57.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.316+0000 7f7f1d3cd640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f180b39c0 con 0x7f7f1809ac40 2024-03-29T11:18:57.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.316+0000 7f7f1d3cd640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:57.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.316+0000 7f7f1d3cd640 10 monclient(hunting): authenticate will time out at 822.478003s 2024-03-29T11:18:57.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.317+0000 7f7f177fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f7f180b4360 0x7f7f1814d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:38694/0 (socket says 172.21.15.94:38694) 2024-03-29T11:18:57.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.317+0000 7f7f177fe640 1 -- 172.21.15.94:0/3649375631 learned_addr learned my addr 172.21.15.94:0/3649375631 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T11:18:57.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.317+0000 7f7f167fc640 1 -- 172.21.15.94:0/3649375631 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1742483439 0 0) 0x7f7f1809da30 con 0x7f7f180b4360 2024-03-29T11:18:57.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.317+0000 7f7f167fc640 10 monclient(hunting): _negotiate 2024-03-29T11:18:57.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.317+0000 7f7f167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:57.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.317+0000 7f7f167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:57.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.317+0000 7f7f167fc640 10 monclient(hunting): my global_id is 4129 2024-03-29T11:18:57.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.317+0000 7f7f167fc640 1 -- 172.21.15.94:0/3649375631 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f04003520 con 0x7f7f180b4360 2024-03-29T11:18:57.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.317+0000 7f7f167fc640 1 -- 172.21.15.94:0/3649375631 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1568901633 0 0) 0x7f7f00001110 con 0x7f7f1809c870 2024-03-29T11:18:57.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.317+0000 7f7f167fc640 10 monclient(hunting): _negotiate 2024-03-29T11:18:57.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.317+0000 7f7f167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:57.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.317+0000 7f7f167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:57.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.317+0000 7f7f167fc640 10 monclient(hunting): my global_id is 4125 2024-03-29T11:18:57.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.317+0000 7f7f167fc640 1 -- 172.21.15.94:0/3649375631 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f1809da30 con 0x7f7f1809c870 2024-03-29T11:18:57.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.317+0000 7f7f167fc640 1 -- 172.21.15.94:0/3649375631 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3055519242 0 0) 0x7f7f180b39c0 con 0x7f7f1809ac40 2024-03-29T11:18:57.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.317+0000 7f7f167fc640 10 monclient(hunting): _negotiate 2024-03-29T11:18:57.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.317+0000 7f7f167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:57.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.317+0000 7f7f167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:57.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.317+0000 7f7f167fc640 10 monclient(hunting): my global_id is 4127 2024-03-29T11:18:57.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.317+0000 7f7f167fc640 1 -- 172.21.15.94:0/3649375631 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f00001110 con 0x7f7f1809ac40 2024-03-29T11:18:57.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.317+0000 7f7f167fc640 1 -- 172.21.15.94:0/3649375631 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1079703749 0 0) 0x7f7f04003520 con 0x7f7f180b4360 2024-03-29T11:18:57.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.317+0000 7f7f167fc640 1 -- 172.21.15.94:0/3649375631 >> v1:172.21.15.17:6790/0 conn(0x7f7f1809ac40 legacy=0x7f7f180b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:57.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.317+0000 7f7f167fc640 1 -- 172.21.15.94:0/3649375631 >> v1:172.21.15.17:6789/0 conn(0x7f7f1809c870 legacy=0x7f7f1813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:57.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.317+0000 7f7f167fc640 10 monclient: _finish_hunting 0 2024-03-29T11:18:57.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.317+0000 7f7f167fc640 1 monclient: found mon.noname-c 2024-03-29T11:18:57.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.317+0000 7f7f167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:57.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.317+0000 7f7f167fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6789/0 2024-03-29T11:18:57.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.317+0000 7f7f167fc640 1 -- 172.21.15.94:0/3649375631 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f180b5160 con 0x7f7f180b4360 2024-03-29T11:18:57.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.317+0000 7f7f167fc640 10 monclient: _finish_auth 0 2024-03-29T11:18:57.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.317+0000 7f7f167fc640 10 monclient: _check_auth_tickets 2024-03-29T11:18:57.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.317+0000 7f7f167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:57.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.317+0000 7f7f167fc640 1 -- 172.21.15.94:0/3649375631 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f7f10001a10 con 0x7f7f180b4360 2024-03-29T11:18:57.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.317+0000 7f7f167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:57.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.317+0000 7f7f167fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:18:57.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.317+0000 7f7f167fc640 10 monclient: dump: 2024-03-29T11:18:57.305 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T11:18:57.305 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:57.305 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.305 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.305 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:57.305 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:57.305 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:57.305 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:57.305 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:57.305 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:57.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.317+0000 7f7f1d3cd640 5 monclient: authenticate success, global_id 4129 2024-03-29T11:18:57.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.317+0000 7f7f1d3cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:57.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.317+0000 7f7f167fc640 1 -- 172.21.15.94:0/3649375631 <== mon.1 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f10001bf0 con 0x7f7f180b4360 2024-03-29T11:18:57.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.317+0000 7f7f167fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:57.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.317+0000 7f7f167fc640 1 -- 172.21.15.94:0/3649375631 <== mon.1 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f7f10002000 con 0x7f7f180b4360 2024-03-29T11:18:57.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.317+0000 7f7f1d3cd640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:57.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.317+0000 7f7f1d3cd640 4 set_mon_vals no callback set 2024-03-29T11:18:57.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.317+0000 7f7f167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:57.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.317+0000 7f7f167fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:18:57.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.317+0000 7f7f167fc640 10 monclient: dump: 2024-03-29T11:18:57.306 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T11:18:57.306 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:57.306 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.306 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.306 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:57.306 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:57.306 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:57.306 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:57.307 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:57.307 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:57.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.318+0000 7f7f1d3cd640 10 monclient: shutdown 2024-03-29T11:18:57.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.318+0000 7f7f1d3cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:57.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.318+0000 7f7f1d3cd640 1 -- 172.21.15.94:0/3649375631 >> v1:172.21.15.94:6789/0 conn(0x7f7f180b4360 legacy=0x7f7f1814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:57.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.318+0000 7f7f1d3cd640 1 -- 172.21.15.94:0/3649375631 shutdown_connections 2024-03-29T11:18:57.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.318+0000 7f7f1d3cd640 1 -- 172.21.15.94:0/3649375631 >> 172.21.15.94:0/3649375631 conn(0x7f7f18066b00 msgr2=0x7f7f18066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:57.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.318+0000 7f7f1d3cd640 1 -- 172.21.15.94:0/3649375631 shutdown_connections 2024-03-29T11:18:57.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.318+0000 7f7f1d3cd640 1 -- 172.21.15.94:0/3649375631 wait complete. 2024-03-29T11:18:57.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.318+0000 7f7f1d3cd640 10 monclient: build_initial_monmap 2024-03-29T11:18:57.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.318+0000 7f7f1d3cd640 10 monclient: monmap: 2024-03-29T11:18:57.307 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-03-29T11:18:57.307 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:57.307 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-03-29T11:18:57.308 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-03-29T11:18:57.308 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:57.308 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:57.308 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:18:57.308 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:18:57.308 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:18:57.308 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:57.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.319+0000 7f7f1d3cd640 1 librados: starting msgr at 2024-03-29T11:18:57.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.319+0000 7f7f1d3cd640 1 librados: starting objecter 2024-03-29T11:18:57.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.319+0000 7f7f1d3cd640 1 Processor -- start 2024-03-29T11:18:57.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.319+0000 7f7f1d3cd640 1 -- start start 2024-03-29T11:18:57.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.319+0000 7f7f1d3cd640 1 librados: setting wanted keys 2024-03-29T11:18:57.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.319+0000 7f7f1d3cd640 1 librados: calling monclient init 2024-03-29T11:18:57.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.319+0000 7f7f1d3cd640 10 monclient: init 2024-03-29T11:18:57.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.319+0000 7f7f1d3cd640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:57.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.319+0000 7f7f1d3cd640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:18:57.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.319+0000 7f7f1d3cd640 10 monclient(hunting): picked mon.noname-a con 0x7f7f1809ac40 addr v1:172.21.15.17:6789/0 2024-03-29T11:18:57.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.319+0000 7f7f1d3cd640 10 monclient(hunting): picked mon.noname-b con 0x7f7f18140a10 addr v1:172.21.15.94:6789/0 2024-03-29T11:18:57.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.319+0000 7f7f1d3cd640 10 monclient(hunting): picked mon.noname-c con 0x7f7f181d7e70 addr v1:172.21.15.17:6790/0 2024-03-29T11:18:57.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.319+0000 7f7f1d3cd640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f181d7280 con 0x7f7f1809ac40 2024-03-29T11:18:57.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.319+0000 7f7f1d3cd640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f18142b20 con 0x7f7f18140a10 2024-03-29T11:18:57.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.319+0000 7f7f1d3cd640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f18144420 con 0x7f7f181d7e70 2024-03-29T11:18:57.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.319+0000 7f7f1d3cd640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:57.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.319+0000 7f7f1d3cd640 10 monclient(hunting): authenticate will time out at 822.480930s 2024-03-29T11:18:57.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.319+0000 7f7f16ffd640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f7f1809ac40 0x7f7f18141690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.94:51484/0 (socket says 172.21.15.94:51484) 2024-03-29T11:18:57.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f16ffd640 1 -- 172.21.15.94:0/393462894 learned_addr learned my addr 172.21.15.94:0/393462894 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T11:18:57.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.319+0000 7f7f0ffff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f7f18140a10 0x7f7f181d7760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:38704/0 (socket says 172.21.15.94:38704) 2024-03-29T11:18:57.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 10 client.?.objecter ms_handle_connect 0x7f7f18140a10 2024-03-29T11:18:57.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:57.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 10 client.?.objecter ms_handle_connect 0x7f7f1809ac40 2024-03-29T11:18:57.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:57.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 10 client.?.objecter ms_handle_connect 0x7f7f181d7e70 2024-03-29T11:18:57.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:57.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 1 -- 172.21.15.94:0/393462894 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4293017710 0 0) 0x7f7f18142b20 con 0x7f7f18140a10 2024-03-29T11:18:57.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:18:57.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:57.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:57.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 10 monclient(hunting): my global_id is 4132 2024-03-29T11:18:57.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 1 -- 172.21.15.94:0/393462894 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ef8004eb0 con 0x7f7f18140a10 2024-03-29T11:18:57.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 1 -- 172.21.15.94:0/393462894 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2029469811 0 0) 0x7f7f181d7280 con 0x7f7f1809ac40 2024-03-29T11:18:57.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:18:57.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:57.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:57.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 10 monclient(hunting): my global_id is 4128 2024-03-29T11:18:57.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 1 -- 172.21.15.94:0/393462894 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f18142b20 con 0x7f7f1809ac40 2024-03-29T11:18:57.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 1 -- 172.21.15.94:0/393462894 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2604193245 0 0) 0x7f7f18144420 con 0x7f7f181d7e70 2024-03-29T11:18:57.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:18:57.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:57.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:57.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 10 monclient(hunting): my global_id is 4130 2024-03-29T11:18:57.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 1 -- 172.21.15.94:0/393462894 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f181d7280 con 0x7f7f181d7e70 2024-03-29T11:18:57.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 1 -- 172.21.15.94:0/393462894 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1348169631 0 0) 0x7f7ef8004eb0 con 0x7f7f18140a10 2024-03-29T11:18:57.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 1 -- 172.21.15.94:0/393462894 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f18144420 con 0x7f7f18140a10 2024-03-29T11:18:57.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 1 -- 172.21.15.94:0/393462894 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f7f000018e0 con 0x7f7f18140a10 2024-03-29T11:18:57.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:57.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:18:57.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 10 monclient(hunting): dump: 2024-03-29T11:18:57.311 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T11:18:57.311 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:57.312 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.312 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.312 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:57.312 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:57.312 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:57.312 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:57.312 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:57.312 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:57.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:57.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 1 -- 172.21.15.94:0/393462894 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4287171265 0 0) 0x7f7f18142b20 con 0x7f7f1809ac40 2024-03-29T11:18:57.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 1 -- 172.21.15.94:0/393462894 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ef8004eb0 con 0x7f7f1809ac40 2024-03-29T11:18:57.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 1 -- 172.21.15.94:0/393462894 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f7f08001970 con 0x7f7f1809ac40 2024-03-29T11:18:57.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:57.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:57.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 10 monclient(hunting): dump: 2024-03-29T11:18:57.313 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T11:18:57.313 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:57.313 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.313 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.313 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:57.313 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:57.313 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:57.313 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:57.313 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:57.313 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:57.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 1 -- 172.21.15.94:0/393462894 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2542612342 0 0) 0x7f7f181d7280 con 0x7f7f181d7e70 2024-03-29T11:18:57.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 1 -- 172.21.15.94:0/393462894 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f18142b20 con 0x7f7f181d7e70 2024-03-29T11:18:57.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 1 -- 172.21.15.94:0/393462894 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f7f10001f00 con 0x7f7f181d7e70 2024-03-29T11:18:57.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:57.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:57.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 10 monclient(hunting): dump: 2024-03-29T11:18:57.314 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T11:18:57.314 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:57.314 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.314 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.314 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:57.314 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:57.314 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:57.314 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:57.314 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:57.314 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:57.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 1 -- 172.21.15.94:0/393462894 <== mon.1 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1467415432 0 0) 0x7f7f18144420 con 0x7f7f18140a10 2024-03-29T11:18:57.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 1 -- 172.21.15.94:0/393462894 >> v1:172.21.15.17:6790/0 conn(0x7f7f181d7e70 legacy=0x7f7f180b0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:57.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 1 -- 172.21.15.94:0/393462894 >> v1:172.21.15.17:6789/0 conn(0x7f7f1809ac40 legacy=0x7f7f18141690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:57.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:18:57.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 1 monclient: found mon.b 2024-03-29T11:18:57.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:57.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:18:57.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 1 -- 172.21.15.94:0/393462894 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f181445b0 con 0x7f7f18140a10 2024-03-29T11:18:57.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:18:57.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:18:57.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:57.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f1d3cd640 5 monclient: authenticate success, global_id 4132 2024-03-29T11:18:57.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f1d3cd640 10 monclient: _renew_subs 2024-03-29T11:18:57.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f1d3cd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:18:57.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f1d3cd640 1 -- 172.21.15.94:0/393462894 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f18140e70 con 0x7f7f18140a10 2024-03-29T11:18:57.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f1d3cd640 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:57.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f1d3cd640 10 monclient: _renew_subs 2024-03-29T11:18:57.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f1d3cd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:18:57.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f1d3cd640 1 -- 172.21.15.94:0/393462894 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f180b1860 con 0x7f7f18140a10 2024-03-29T11:18:57.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 1 -- 172.21.15.94:0/393462894 <== mon.1 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f00003610 con 0x7f7f18140a10 2024-03-29T11:18:57.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f1d3cd640 1 librados: init done 2024-03-29T11:18:57.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:57.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 1 -- 172.21.15.94:0/393462894 <== mon.1 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f7f00003800 con 0x7f7f18140a10 2024-03-29T11:18:57.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:57.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:18:57.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 10 monclient: dump: 2024-03-29T11:18:57.316 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T11:18:57.316 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:57.316 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.316 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.316 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:57.317 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:57.317 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:57.317 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:57.317 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:57.317 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:57.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 1 -- 172.21.15.94:0/393462894 <== mon.1 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f7f00003db0 con 0x7f7f18140a10 2024-03-29T11:18:57.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f15ffb640 4 set_mon_vals no callback set 2024-03-29T11:18:57.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 1 -- 172.21.15.94:0/393462894 <== mon.1 v1:172.21.15.94:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 2399094797 0 0) 0x7f7f00006440 con 0x7f7f18140a10 2024-03-29T11:18:57.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 10 client.4132.objecter ms_dispatch 0x7f7f18000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T11:18:57.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 3 client.4132.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T11:18:57.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 3 client.4132.objecter handle_osd_map decoding full epoch 4 2024-03-29T11:18:57.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.320+0000 7f7f0f7fe640 20 client.4132.objecter dump_active .. 0 homeless 2024-03-29T11:18:57.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.321+0000 7f7f1d3cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:57.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.321+0000 7f7f1d3cd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:57.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.321+0000 7f7f1d3cd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:18:57.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.321+0000 7f7f1d3cd640 1 -- 172.21.15.94:0/393462894 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f18143220 con 0x7f7f18140a10 2024-03-29T11:18:57.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.323+0000 7f7f0f7fe640 1 -- 172.21.15.94:0/393462894 <== mon.1 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f7f00006020 con 0x7f7f18140a10 2024-03-29T11:18:57.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.323+0000 7f7f0f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:57.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.323+0000 7f7f0f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:57.350 INFO:tasks.ceph.mgr.y.smithi017.stderr:2024-03-29T11:18:57.373+0000 7f6f3d22e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:18:57.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.396+0000 7f7f1d3cd640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b2fd6920-a919-4184-bce2-1aee9c71f90e", "id": 2}] 2024-03-29T11:18:57.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.396+0000 7f7f1d3cd640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b2fd6920-a919-4184-bce2-1aee9c71f90e", "id": 2}] 2024-03-29T11:18:57.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.396+0000 7f7f1d3cd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:18:57.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.396+0000 7f7f1d3cd640 1 -- 172.21.15.94:0/393462894 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd new", "uuid": "b2fd6920-a919-4184-bce2-1aee9c71f90e", "id": 2} v 0) v1 -- 0x7f7f180b1b10 con 0x7f7f18140a10 2024-03-29T11:18:57.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.399+0000 7f7f0f7fe640 1 -- 172.21.15.94:0/393462894 <== mon.1 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b2fd6920-a919-4184-bce2-1aee9c71f90e", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 1307705428 0 216599125) 0x7f7f000062a0 con 0x7f7f18140a10 2024-03-29T11:18:57.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.399+0000 7f7f0f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b2fd6920-a919-4184-bce2-1aee9c71f90e", "id": 2}] 2024-03-29T11:18:57.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.399+0000 7f7f0f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:57.376 INFO:teuthology.orchestra.run.smithi094.stdout:2 2024-03-29T11:18:57.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.400+0000 7f7f1d3cd640 10 librados: watch_flush enter 2024-03-29T11:18:57.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.400+0000 7f7f1d3cd640 10 librados: watch_flush exit 2024-03-29T11:18:57.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.400+0000 7f7f1d3cd640 20 client.4132.objecter shutdown clearing up homeless session... 2024-03-29T11:18:57.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.400+0000 7f7f1d3cd640 10 client.4132.objecter successfully canceled tick 2024-03-29T11:18:57.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.400+0000 7f7f1d3cd640 10 monclient: shutdown 2024-03-29T11:18:57.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.400+0000 7f7f1d3cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:57.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.400+0000 7f7f1d3cd640 1 -- 172.21.15.94:0/393462894 >> v1:172.21.15.94:6789/0 conn(0x7f7f18140a10 legacy=0x7f7f181d7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:57.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.400+0000 7f7f1d3cd640 1 -- 172.21.15.94:0/393462894 shutdown_connections 2024-03-29T11:18:57.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.400+0000 7f7f1d3cd640 1 -- 172.21.15.94:0/393462894 >> 172.21.15.94:0/393462894 conn(0x7f7f18066b00 msgr2=0x7f7f1813d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:57.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.400+0000 7f7f1d3cd640 1 -- 172.21.15.94:0/393462894 shutdown_connections 2024-03-29T11:18:57.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.400+0000 7f7f1d3cd640 1 -- 172.21.15.94:0/393462894 wait complete. 2024-03-29T11:18:57.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.400+0000 7f7f1d3cd640 1 librados: shutdown 2024-03-29T11:18:57.387 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster ceph osd new cded4702-dbf5-48f9-9d2c-2b49173dee68 3 2024-03-29T11:18:57.397 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-03-29T11:18:57.421+0000 7fcdfee3c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:18:57.406 INFO:tasks.ceph.mgr.y.smithi017.stderr:2024-03-29T11:18:57.429+0000 7f6f3d22e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:18:57.452 INFO:tasks.ceph.mgr.y.smithi017.stderr:2024-03-29T11:18:57.475+0000 7f6f3d22e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:18:57.455 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-03-29T11:18:57.478+0000 7fcdfee3c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:18:57.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.509+0000 7f9ff1da8640 10 monclient: get_monmap_and_config 2024-03-29T11:18:57.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.509+0000 7f9ff1da8640 10 monclient: build_initial_monmap 2024-03-29T11:18:57.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.509+0000 7f9ff1da8640 10 monclient: monmap: 2024-03-29T11:18:57.487 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-03-29T11:18:57.487 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:57.487 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:57.510740+0000 2024-03-29T11:18:57.487 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:57.510740+0000 2024-03-29T11:18:57.487 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:57.487 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:57.487 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:18:57.487 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:18:57.487 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:18:57.487 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:57.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.510+0000 7f9ff1da8640 1 Processor -- start 2024-03-29T11:18:57.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.510+0000 7f9ff1da8640 1 -- start start 2024-03-29T11:18:57.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.510+0000 7f9ff1da8640 10 monclient: init 2024-03-29T11:18:57.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9ff1da8640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:57.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9ff1da8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:18:57.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9ff1da8640 10 monclient(hunting): picked mon.noname-a con 0x7f9fec09ac40 addr v1:172.21.15.17:6789/0 2024-03-29T11:18:57.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9ff1da8640 10 monclient(hunting): picked mon.noname-c con 0x7f9fec09c870 addr v1:172.21.15.94:6789/0 2024-03-29T11:18:57.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9ff1da8640 10 monclient(hunting): picked mon.noname-b con 0x7f9fec0b4360 addr v1:172.21.15.17:6790/0 2024-03-29T11:18:57.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9ff1da8640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fec09dec0 con 0x7f9fec09ac40 2024-03-29T11:18:57.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9ff1da8640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fec09da30 con 0x7f9fec09c870 2024-03-29T11:18:57.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9ff1da8640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fec0b39c0 con 0x7f9fec0b4360 2024-03-29T11:18:57.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9ff1da8640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:57.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9ff1da8640 10 monclient(hunting): authenticate will time out at 822.672255s 2024-03-29T11:18:57.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9feaffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f9fec09c870 0x7f9fec13f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:38718/0 (socket says 172.21.15.94:38718) 2024-03-29T11:18:57.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9feaffd640 1 -- 172.21.15.94:0/2253591960 learned_addr learned my addr 172.21.15.94:0/2253591960 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T11:18:57.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9fea7fc640 1 -- 172.21.15.94:0/2253591960 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 438715993 0 0) 0x7f9fec09da30 con 0x7f9fec09c870 2024-03-29T11:18:57.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9fea7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:18:57.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9fea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:57.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9fea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:57.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9fea7fc640 10 monclient(hunting): my global_id is 4135 2024-03-29T11:18:57.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9fea7fc640 1 -- 172.21.15.94:0/2253591960 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fd4003520 con 0x7f9fec09c870 2024-03-29T11:18:57.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9fea7fc640 1 -- 172.21.15.94:0/2253591960 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1009033690 0 0) 0x7f9fe0001110 con 0x7f9fec09ac40 2024-03-29T11:18:57.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9fea7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:18:57.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9fea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:57.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9fea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:57.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9fea7fc640 10 monclient(hunting): my global_id is 4131 2024-03-29T11:18:57.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9fea7fc640 1 -- 172.21.15.94:0/2253591960 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fec09da30 con 0x7f9fec09ac40 2024-03-29T11:18:57.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9fea7fc640 1 -- 172.21.15.94:0/2253591960 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3658065471 0 0) 0x7f9fec0b39c0 con 0x7f9fec0b4360 2024-03-29T11:18:57.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9fea7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:18:57.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9fea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:57.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9fea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:57.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9fea7fc640 10 monclient(hunting): my global_id is 4133 2024-03-29T11:18:57.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9fea7fc640 1 -- 172.21.15.94:0/2253591960 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fe0001110 con 0x7f9fec0b4360 2024-03-29T11:18:57.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9fea7fc640 1 -- 172.21.15.94:0/2253591960 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1855779883 0 0) 0x7f9fd4003520 con 0x7f9fec09c870 2024-03-29T11:18:57.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9fea7fc640 1 -- 172.21.15.94:0/2253591960 >> v1:172.21.15.17:6790/0 conn(0x7f9fec0b4360 legacy=0x7f9fec14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:57.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9fea7fc640 1 -- 172.21.15.94:0/2253591960 >> v1:172.21.15.17:6789/0 conn(0x7f9fec09ac40 legacy=0x7f9fec0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:57.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9fea7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:18:57.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9fea7fc640 1 monclient: found mon.noname-c 2024-03-29T11:18:57.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9fea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:57.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9fea7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6789/0 2024-03-29T11:18:57.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9fea7fc640 1 -- 172.21.15.94:0/2253591960 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fec0b5160 con 0x7f9fec09c870 2024-03-29T11:18:57.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9fea7fc640 10 monclient: _finish_auth 0 2024-03-29T11:18:57.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9fea7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:18:57.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9fea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:57.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9ff1da8640 5 monclient: authenticate success, global_id 4135 2024-03-29T11:18:57.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9ff1da8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:57.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9fea7fc640 1 -- 172.21.15.94:0/2253591960 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f9fdc001690 con 0x7f9fec09c870 2024-03-29T11:18:57.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9fea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:57.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9fea7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:18:57.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9fea7fc640 10 monclient: dump: 2024-03-29T11:18:57.499 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T11:18:57.499 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:57.499 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.499 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.499 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:57.499 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:57.499 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:57.499 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:57.499 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:57.499 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:57.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9ff1da8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:57.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.511+0000 7f9fea7fc640 1 -- 172.21.15.94:0/2253591960 <== mon.1 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9fdc001870 con 0x7f9fec09c870 2024-03-29T11:18:57.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.512+0000 7f9fea7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:57.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.512+0000 7f9fea7fc640 1 -- 172.21.15.94:0/2253591960 <== mon.1 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f9fdc001c80 con 0x7f9fec09c870 2024-03-29T11:18:57.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.512+0000 7f9ff1da8640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:57.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.512+0000 7f9ff1da8640 4 set_mon_vals no callback set 2024-03-29T11:18:57.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.512+0000 7f9fea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:57.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.512+0000 7f9fea7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:18:57.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.512+0000 7f9fea7fc640 10 monclient: dump: 2024-03-29T11:18:57.500 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T11:18:57.500 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:57.500 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.500 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.500 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:57.500 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:57.500 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:57.500 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:57.500 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:57.500 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:57.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.512+0000 7f9ff1da8640 10 monclient: shutdown 2024-03-29T11:18:57.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.512+0000 7f9ff1da8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:57.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.512+0000 7f9ff1da8640 1 -- 172.21.15.94:0/2253591960 >> v1:172.21.15.94:6789/0 conn(0x7f9fec09c870 legacy=0x7f9fec13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:57.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.512+0000 7f9ff1da8640 1 -- 172.21.15.94:0/2253591960 shutdown_connections 2024-03-29T11:18:57.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.512+0000 7f9ff1da8640 1 -- 172.21.15.94:0/2253591960 >> 172.21.15.94:0/2253591960 conn(0x7f9fec066b00 msgr2=0x7f9fec066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:57.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.512+0000 7f9ff1da8640 1 -- 172.21.15.94:0/2253591960 shutdown_connections 2024-03-29T11:18:57.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.512+0000 7f9ff1da8640 1 -- 172.21.15.94:0/2253591960 wait complete. 2024-03-29T11:18:57.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.513+0000 7f9ff1da8640 10 monclient: build_initial_monmap 2024-03-29T11:18:57.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.513+0000 7f9ff1da8640 10 monclient: monmap: 2024-03-29T11:18:57.501 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-03-29T11:18:57.501 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:57.501 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-03-29T11:18:57.501 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-03-29T11:18:57.501 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:57.501 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:57.502 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:18:57.502 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:18:57.502 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:18:57.502 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:57.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.513+0000 7f9ff1da8640 1 librados: starting msgr at 2024-03-29T11:18:57.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.513+0000 7f9ff1da8640 1 librados: starting objecter 2024-03-29T11:18:57.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.513+0000 7f9ff1da8640 1 Processor -- start 2024-03-29T11:18:57.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.513+0000 7f9ff1da8640 1 -- start start 2024-03-29T11:18:57.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.513+0000 7f9ff1da8640 1 librados: setting wanted keys 2024-03-29T11:18:57.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.513+0000 7f9ff1da8640 1 librados: calling monclient init 2024-03-29T11:18:57.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.513+0000 7f9ff1da8640 10 monclient: init 2024-03-29T11:18:57.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9ff1da8640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:57.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9ff1da8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:18:57.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9ff1da8640 10 monclient(hunting): picked mon.noname-c con 0x7f9fec09ac40 addr v1:172.21.15.17:6790/0 2024-03-29T11:18:57.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9ff1da8640 10 monclient(hunting): picked mon.noname-b con 0x7f9fec1d7bc0 addr v1:172.21.15.94:6789/0 2024-03-29T11:18:57.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9ff1da8640 10 monclient(hunting): picked mon.noname-a con 0x7f9fec0b17f0 addr v1:172.21.15.17:6789/0 2024-03-29T11:18:57.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9ff1da8640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fec13b890 con 0x7f9fec0b17f0 2024-03-29T11:18:57.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9ff1da8640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fec1d76a0 con 0x7f9fec1d7bc0 2024-03-29T11:18:57.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9ff1da8640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fec1406b0 con 0x7f9fec09ac40 2024-03-29T11:18:57.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9ff1da8640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:57.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9ff1da8640 10 monclient(hunting): authenticate will time out at 822.675352s 2024-03-29T11:18:57.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9feaffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f9fec1d7bc0 0x7f9fec13aa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:38734/0 (socket says 172.21.15.94:38734) 2024-03-29T11:18:57.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9feaffd640 1 -- 172.21.15.94:0/3742417987 learned_addr learned my addr 172.21.15.94:0/3742417987 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T11:18:57.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9fd3fff640 10 client.?.objecter ms_handle_connect 0x7f9fec1d7bc0 2024-03-29T11:18:57.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9fd3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:57.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9fd3fff640 10 client.?.objecter ms_handle_connect 0x7f9fec0b17f0 2024-03-29T11:18:57.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9fd3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:57.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9fd3fff640 10 client.?.objecter ms_handle_connect 0x7f9fec09ac40 2024-03-29T11:18:57.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9fd3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:57.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9fd3fff640 1 -- 172.21.15.94:0/3742417987 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4285657622 0 0) 0x7f9fec1d76a0 con 0x7f9fec1d7bc0 2024-03-29T11:18:57.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9fd3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:18:57.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9fd3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:57.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9fd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:57.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9fd3fff640 10 monclient(hunting): my global_id is 4138 2024-03-29T11:18:57.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9fd3fff640 1 -- 172.21.15.94:0/3742417987 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fd8004eb0 con 0x7f9fec1d7bc0 2024-03-29T11:18:57.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9fd3fff640 1 -- 172.21.15.94:0/3742417987 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 731723202 0 0) 0x7f9fec13b890 con 0x7f9fec0b17f0 2024-03-29T11:18:57.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9fd3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:18:57.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9fd3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:57.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9fd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:57.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9fd3fff640 10 monclient(hunting): my global_id is 4134 2024-03-29T11:18:57.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9fd3fff640 1 -- 172.21.15.94:0/3742417987 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fec1d76a0 con 0x7f9fec0b17f0 2024-03-29T11:18:57.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9fd3fff640 1 -- 172.21.15.94:0/3742417987 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1768698616 0 0) 0x7f9fec1406b0 con 0x7f9fec09ac40 2024-03-29T11:18:57.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9fd3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:18:57.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9fd3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:57.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9fd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:57.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9fd3fff640 10 monclient(hunting): my global_id is 4136 2024-03-29T11:18:57.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9fd3fff640 1 -- 172.21.15.94:0/3742417987 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fec13b890 con 0x7f9fec09ac40 2024-03-29T11:18:57.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9fd3fff640 1 -- 172.21.15.94:0/3742417987 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1155850407 0 0) 0x7f9fd8004eb0 con 0x7f9fec1d7bc0 2024-03-29T11:18:57.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9fd3fff640 1 -- 172.21.15.94:0/3742417987 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9fec1406b0 con 0x7f9fec1d7bc0 2024-03-29T11:18:57.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9fd3fff640 1 -- 172.21.15.94:0/3742417987 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f9fdc001bc0 con 0x7f9fec1d7bc0 2024-03-29T11:18:57.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9fd3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:57.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9fd3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:18:57.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9fd3fff640 10 monclient(hunting): dump: 2024-03-29T11:18:57.505 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T11:18:57.505 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:57.505 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.505 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.506 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:57.506 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:57.506 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:57.506 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:57.506 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:57.506 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:57.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9fd3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:57.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9fd3fff640 1 -- 172.21.15.94:0/3742417987 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 993951257 0 0) 0x7f9fec1d76a0 con 0x7f9fec0b17f0 2024-03-29T11:18:57.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9fd3fff640 1 -- 172.21.15.94:0/3742417987 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9fd8004eb0 con 0x7f9fec0b17f0 2024-03-29T11:18:57.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9fd3fff640 1 -- 172.21.15.94:0/3742417987 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f9fe4001b50 con 0x7f9fec0b17f0 2024-03-29T11:18:57.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9fd3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:57.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9fd3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:57.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.514+0000 7f9fd3fff640 10 monclient(hunting): dump: 2024-03-29T11:18:57.506 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T11:18:57.507 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:57.507 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.507 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.507 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:57.507 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:57.507 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:57.507 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:57.507 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:57.507 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:57.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.515+0000 7f9fd3fff640 1 -- 172.21.15.94:0/3742417987 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2250295623 0 0) 0x7f9fec13b890 con 0x7f9fec09ac40 2024-03-29T11:18:57.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.515+0000 7f9fd3fff640 1 -- 172.21.15.94:0/3742417987 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9fec1d76a0 con 0x7f9fec09ac40 2024-03-29T11:18:57.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.515+0000 7f9fd3fff640 1 -- 172.21.15.94:0/3742417987 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f9fe00018a0 con 0x7f9fec09ac40 2024-03-29T11:18:57.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.515+0000 7f9fd3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:57.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.515+0000 7f9fd3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:57.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.515+0000 7f9fd3fff640 10 monclient(hunting): dump: 2024-03-29T11:18:57.508 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T11:18:57.508 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:57.508 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.508 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.508 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:57.508 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:57.508 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:57.508 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:57.508 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:57.508 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:57.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.515+0000 7f9fd3fff640 1 -- 172.21.15.94:0/3742417987 <== mon.1 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1485381509 0 0) 0x7f9fec1406b0 con 0x7f9fec1d7bc0 2024-03-29T11:18:57.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.515+0000 7f9fd3fff640 1 -- 172.21.15.94:0/3742417987 >> v1:172.21.15.17:6790/0 conn(0x7f9fec09ac40 legacy=0x7f9fec13ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:57.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.515+0000 7f9fd3fff640 1 -- 172.21.15.94:0/3742417987 >> v1:172.21.15.17:6789/0 conn(0x7f9fec0b17f0 legacy=0x7f9fec13b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:57.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.515+0000 7f9fd3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:18:57.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.515+0000 7f9fd3fff640 1 monclient: found mon.b 2024-03-29T11:18:57.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.515+0000 7f9fd3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:57.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.515+0000 7f9fd3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:18:57.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.515+0000 7f9fd3fff640 1 -- 172.21.15.94:0/3742417987 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fec140840 con 0x7f9fec1d7bc0 2024-03-29T11:18:57.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.515+0000 7f9fd3fff640 10 monclient: _finish_auth 0 2024-03-29T11:18:57.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.515+0000 7f9fd3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:18:57.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.515+0000 7f9fd3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:57.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.515+0000 7f9ff1da8640 5 monclient: authenticate success, global_id 4138 2024-03-29T11:18:57.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.515+0000 7f9ff1da8640 10 monclient: _renew_subs 2024-03-29T11:18:57.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.515+0000 7f9ff1da8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:18:57.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.515+0000 7f9ff1da8640 1 -- 172.21.15.94:0/3742417987 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fec0b6f40 con 0x7f9fec1d7bc0 2024-03-29T11:18:57.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.515+0000 7f9ff1da8640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:57.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.515+0000 7f9ff1da8640 10 monclient: _renew_subs 2024-03-29T11:18:57.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.515+0000 7f9ff1da8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:18:57.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.515+0000 7f9ff1da8640 1 -- 172.21.15.94:0/3742417987 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fec1d6f40 con 0x7f9fec1d7bc0 2024-03-29T11:18:57.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.515+0000 7f9fd3fff640 1 -- 172.21.15.94:0/3742417987 <== mon.1 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9fdc004bf0 con 0x7f9fec1d7bc0 2024-03-29T11:18:57.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.515+0000 7f9ff1da8640 1 librados: init done 2024-03-29T11:18:57.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.515+0000 7f9fd3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:57.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.515+0000 7f9fd3fff640 1 -- 172.21.15.94:0/3742417987 <== mon.1 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f9fdc004f00 con 0x7f9fec1d7bc0 2024-03-29T11:18:57.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.515+0000 7f9fd3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:57.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.515+0000 7f9fd3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:18:57.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.515+0000 7f9fd3fff640 10 monclient: dump: 2024-03-29T11:18:57.510 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T11:18:57.510 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:57.510 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.510 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.510 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:57.510 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:57.510 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:57.510 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:57.511 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:57.511 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:57.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.515+0000 7f9fd3fff640 1 -- 172.21.15.94:0/3742417987 <== mon.1 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9fdc005470 con 0x7f9fec1d7bc0 2024-03-29T11:18:57.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.515+0000 7f9fe9ffb640 4 set_mon_vals no callback set 2024-03-29T11:18:57.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.515+0000 7f9fd3fff640 1 -- 172.21.15.94:0/3742417987 <== mon.1 v1:172.21.15.94:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 4215942542 0 0) 0x7f9fdc005bb0 con 0x7f9fec1d7bc0 2024-03-29T11:18:57.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.515+0000 7f9fd3fff640 10 client.4138.objecter ms_dispatch 0x7f9fec000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T11:18:57.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.515+0000 7f9fd3fff640 3 client.4138.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T11:18:57.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.515+0000 7f9fd3fff640 3 client.4138.objecter handle_osd_map decoding full epoch 5 2024-03-29T11:18:57.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.515+0000 7f9fd3fff640 20 client.4138.objecter dump_active .. 0 homeless 2024-03-29T11:18:57.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.516+0000 7f9ff1da8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:57.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.516+0000 7f9ff1da8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:57.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.516+0000 7f9ff1da8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:18:57.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.516+0000 7f9ff1da8640 1 -- 172.21.15.94:0/3742417987 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fec09b020 con 0x7f9fec1d7bc0 2024-03-29T11:18:57.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.518+0000 7f9fd3fff640 1 -- 172.21.15.94:0/3742417987 <== mon.1 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9fdc005e90 con 0x7f9fec1d7bc0 2024-03-29T11:18:57.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.518+0000 7f9fd3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:57.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.518+0000 7f9fd3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:57.512 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-03-29T11:18:57.526+0000 7fcdfee3c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:18:57.512 INFO:tasks.ceph.mgr.y.smithi017.stderr:2024-03-29T11:18:57.531+0000 7f6f3d22e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:18:57.552 INFO:tasks.ceph.mgr.y.smithi017.stderr:2024-03-29T11:18:57.575+0000 7f6f3d22e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:18:57.558 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-03-29T11:18:57.581+0000 7fcdfee3c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:18:57.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.591+0000 7f9ff1da8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "cded4702-dbf5-48f9-9d2c-2b49173dee68", "id": 3}] 2024-03-29T11:18:57.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.591+0000 7f9ff1da8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "cded4702-dbf5-48f9-9d2c-2b49173dee68", "id": 3}] 2024-03-29T11:18:57.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.591+0000 7f9ff1da8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:18:57.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.591+0000 7f9ff1da8640 1 -- 172.21.15.94:0/3742417987 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd new", "uuid": "cded4702-dbf5-48f9-9d2c-2b49173dee68", "id": 3} v 0) v1 -- 0x7f9fec04f730 con 0x7f9fec1d7bc0 2024-03-29T11:18:57.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.593+0000 7f9fd3fff640 1 -- 172.21.15.94:0/3742417987 <== mon.1 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cded4702-dbf5-48f9-9d2c-2b49173dee68", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 2407188139 0 525046306) 0x7f9fdc005720 con 0x7f9fec1d7bc0 2024-03-29T11:18:57.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.593+0000 7f9fd3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "cded4702-dbf5-48f9-9d2c-2b49173dee68", "id": 3}] 2024-03-29T11:18:57.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.593+0000 7f9fd3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:57.571 INFO:teuthology.orchestra.run.smithi094.stdout:3 2024-03-29T11:18:57.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.594+0000 7f9ff1da8640 10 librados: watch_flush enter 2024-03-29T11:18:57.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.594+0000 7f9ff1da8640 10 librados: watch_flush exit 2024-03-29T11:18:57.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.594+0000 7f9ff1da8640 20 client.4138.objecter shutdown clearing up homeless session... 2024-03-29T11:18:57.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.594+0000 7f9ff1da8640 10 client.4138.objecter successfully canceled tick 2024-03-29T11:18:57.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.594+0000 7f9ff1da8640 10 monclient: shutdown 2024-03-29T11:18:57.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.594+0000 7f9ff1da8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:57.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.594+0000 7f9ff1da8640 1 -- 172.21.15.94:0/3742417987 >> v1:172.21.15.94:6789/0 conn(0x7f9fec1d7bc0 legacy=0x7f9fec13aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:57.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.594+0000 7f9ff1da8640 1 -- 172.21.15.94:0/3742417987 shutdown_connections 2024-03-29T11:18:57.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.594+0000 7f9ff1da8640 1 -- 172.21.15.94:0/3742417987 >> 172.21.15.94:0/3742417987 conn(0x7f9fec066b00 msgr2=0x7f9fec13d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:57.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.594+0000 7f9ff1da8640 1 -- 172.21.15.94:0/3742417987 shutdown_connections 2024-03-29T11:18:57.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.594+0000 7f9ff1da8640 1 -- 172.21.15.94:0/3742417987 wait complete. 2024-03-29T11:18:57.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.594+0000 7f9ff1da8640 1 librados: shutdown 2024-03-29T11:18:57.581 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster ceph osd new f43c11bd-586c-4992-bb35-04023637a513 4 2024-03-29T11:18:57.603 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-03-29T11:18:57.626+0000 7fcdfee3c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:18:57.672 INFO:tasks.ceph.mgr.y.smithi017.stderr:2024-03-29T11:18:57.695+0000 7f6f3d22e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:18:57.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.726+0000 7f46f1e33640 10 monclient: get_monmap_and_config 2024-03-29T11:18:57.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.726+0000 7f46f1e33640 10 monclient: build_initial_monmap 2024-03-29T11:18:57.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.726+0000 7f46f1e33640 10 monclient: monmap: 2024-03-29T11:18:57.703 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-03-29T11:18:57.703 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:57.703 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:57.727585+0000 2024-03-29T11:18:57.704 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:57.727585+0000 2024-03-29T11:18:57.704 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:57.704 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:57.704 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:18:57.704 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:18:57.704 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:18:57.704 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:57.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.727+0000 7f46f1e33640 1 Processor -- start 2024-03-29T11:18:57.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.727+0000 7f46f1e33640 1 -- start start 2024-03-29T11:18:57.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.727+0000 7f46f1e33640 10 monclient: init 2024-03-29T11:18:57.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.727+0000 7f46f1e33640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:57.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.727+0000 7f46f1e33640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:18:57.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.727+0000 7f46f1e33640 10 monclient(hunting): picked mon.noname-a con 0x7f46ec09ac40 addr v1:172.21.15.17:6789/0 2024-03-29T11:18:57.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.727+0000 7f46f1e33640 10 monclient(hunting): picked mon.noname-b con 0x7f46ec09c870 addr v1:172.21.15.17:6790/0 2024-03-29T11:18:57.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46f1e33640 10 monclient(hunting): picked mon.noname-c con 0x7f46ec0b4360 addr v1:172.21.15.94:6789/0 2024-03-29T11:18:57.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46f1e33640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46ec09dec0 con 0x7f46ec09ac40 2024-03-29T11:18:57.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46f1e33640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46ec09da30 con 0x7f46ec0b4360 2024-03-29T11:18:57.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46f1e33640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46ec0b39c0 con 0x7f46ec09c870 2024-03-29T11:18:57.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46f1e33640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:57.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46f1e33640 10 monclient(hunting): authenticate will time out at 822.889081s 2024-03-29T11:18:57.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46ebfff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f46ec0b4360 0x7f46ec14d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:53528/0 (socket says 172.21.15.94:53528) 2024-03-29T11:18:57.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46ebfff640 1 -- 172.21.15.94:0/75128084 learned_addr learned my addr 172.21.15.94:0/75128084 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T11:18:57.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46ea7fc640 1 -- 172.21.15.94:0/75128084 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2100619950 0 0) 0x7f46ec09da30 con 0x7f46ec0b4360 2024-03-29T11:18:57.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46ea7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:18:57.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46ea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:57.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:57.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46ea7fc640 10 monclient(hunting): my global_id is 4141 2024-03-29T11:18:57.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46ea7fc640 1 -- 172.21.15.94:0/75128084 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46d4003520 con 0x7f46ec0b4360 2024-03-29T11:18:57.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46ea7fc640 1 -- 172.21.15.94:0/75128084 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1360602463 0 0) 0x7f46dc001490 con 0x7f46ec09ac40 2024-03-29T11:18:57.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46ea7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:18:57.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46ea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:57.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:57.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46ea7fc640 10 monclient(hunting): my global_id is 4137 2024-03-29T11:18:57.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46ea7fc640 1 -- 172.21.15.94:0/75128084 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46ec09da30 con 0x7f46ec09ac40 2024-03-29T11:18:57.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46ea7fc640 1 -- 172.21.15.94:0/75128084 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3196634317 0 0) 0x7f46ec0b39c0 con 0x7f46ec09c870 2024-03-29T11:18:57.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46ea7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:18:57.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46ea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:57.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:57.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46ea7fc640 10 monclient(hunting): my global_id is 4139 2024-03-29T11:18:57.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46ea7fc640 1 -- 172.21.15.94:0/75128084 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46dc001490 con 0x7f46ec09c870 2024-03-29T11:18:57.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46ea7fc640 1 -- 172.21.15.94:0/75128084 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3124338227 0 0) 0x7f46d4003520 con 0x7f46ec0b4360 2024-03-29T11:18:57.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46ea7fc640 1 -- 172.21.15.94:0/75128084 >> v1:172.21.15.17:6790/0 conn(0x7f46ec09c870 legacy=0x7f46ec13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:57.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46ea7fc640 1 -- 172.21.15.94:0/75128084 >> v1:172.21.15.17:6789/0 conn(0x7f46ec09ac40 legacy=0x7f46ec0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:57.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46ea7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:18:57.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46ea7fc640 1 monclient: found mon.noname-c 2024-03-29T11:18:57.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46ea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:57.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46ea7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6789/0 2024-03-29T11:18:57.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46ea7fc640 1 -- 172.21.15.94:0/75128084 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46ec0b5160 con 0x7f46ec0b4360 2024-03-29T11:18:57.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46ea7fc640 10 monclient: _finish_auth 0 2024-03-29T11:18:57.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46ea7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:18:57.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:57.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46ea7fc640 1 -- 172.21.15.94:0/75128084 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f46e4001690 con 0x7f46ec0b4360 2024-03-29T11:18:57.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46f1e33640 5 monclient: authenticate success, global_id 4141 2024-03-29T11:18:57.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46f1e33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:57.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:57.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46ea7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:18:57.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46ea7fc640 10 monclient: dump: 2024-03-29T11:18:57.715 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T11:18:57.715 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:57.715 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.715 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.715 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:57.715 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:57.715 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:57.715 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:57.715 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:57.716 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:57.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46f1e33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:57.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46ea7fc640 1 -- 172.21.15.94:0/75128084 <== mon.1 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46e4001870 con 0x7f46ec0b4360 2024-03-29T11:18:57.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46ea7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:57.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46f1e33640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:57.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46ea7fc640 1 -- 172.21.15.94:0/75128084 <== mon.1 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f46e4001c80 con 0x7f46ec0b4360 2024-03-29T11:18:57.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46f1e33640 4 set_mon_vals no callback set 2024-03-29T11:18:57.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:57.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46ea7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:18:57.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.728+0000 7f46ea7fc640 10 monclient: dump: 2024-03-29T11:18:57.716 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T11:18:57.716 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:57.716 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.716 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.716 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:57.717 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:57.717 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:57.717 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:57.717 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:57.717 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:57.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.729+0000 7f46f1e33640 10 monclient: shutdown 2024-03-29T11:18:57.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.729+0000 7f46f1e33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:57.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.729+0000 7f46f1e33640 1 -- 172.21.15.94:0/75128084 >> v1:172.21.15.94:6789/0 conn(0x7f46ec0b4360 legacy=0x7f46ec14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:57.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.729+0000 7f46f1e33640 1 -- 172.21.15.94:0/75128084 shutdown_connections 2024-03-29T11:18:57.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.729+0000 7f46f1e33640 1 -- 172.21.15.94:0/75128084 >> 172.21.15.94:0/75128084 conn(0x7f46ec066b00 msgr2=0x7f46ec066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:57.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.729+0000 7f46f1e33640 1 -- 172.21.15.94:0/75128084 shutdown_connections 2024-03-29T11:18:57.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.729+0000 7f46f1e33640 1 -- 172.21.15.94:0/75128084 wait complete. 2024-03-29T11:18:57.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.729+0000 7f46f1e33640 10 monclient: build_initial_monmap 2024-03-29T11:18:57.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.729+0000 7f46f1e33640 10 monclient: monmap: 2024-03-29T11:18:57.717 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-03-29T11:18:57.718 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:57.718 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-03-29T11:18:57.718 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-03-29T11:18:57.718 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:57.718 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:57.718 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:18:57.718 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:18:57.718 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:18:57.718 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:57.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.730+0000 7f46f1e33640 1 librados: starting msgr at 2024-03-29T11:18:57.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.730+0000 7f46f1e33640 1 librados: starting objecter 2024-03-29T11:18:57.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.730+0000 7f46f1e33640 1 Processor -- start 2024-03-29T11:18:57.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.730+0000 7f46f1e33640 1 -- start start 2024-03-29T11:18:57.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.730+0000 7f46f1e33640 1 librados: setting wanted keys 2024-03-29T11:18:57.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.730+0000 7f46f1e33640 1 librados: calling monclient init 2024-03-29T11:18:57.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.730+0000 7f46f1e33640 10 monclient: init 2024-03-29T11:18:57.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.730+0000 7f46f1e33640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:57.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.730+0000 7f46f1e33640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:18:57.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46f1e33640 10 monclient(hunting): picked mon.noname-b con 0x7f46ec09ac40 addr v1:172.21.15.94:6789/0 2024-03-29T11:18:57.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46f1e33640 10 monclient(hunting): picked mon.noname-c con 0x7f46ec140cc0 addr v1:172.21.15.17:6790/0 2024-03-29T11:18:57.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46f1e33640 10 monclient(hunting): picked mon.noname-a con 0x7f46ec139420 addr v1:172.21.15.17:6789/0 2024-03-29T11:18:57.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46f1e33640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46ec141cf0 con 0x7f46ec139420 2024-03-29T11:18:57.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46f1e33640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46ec0afb40 con 0x7f46ec09ac40 2024-03-29T11:18:57.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46f1e33640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46ec139890 con 0x7f46ec140cc0 2024-03-29T11:18:57.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46f1e33640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:57.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46f1e33640 10 monclient(hunting): authenticate will time out at 822.892106s 2024-03-29T11:18:57.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46eb7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f46ec09ac40 0x7f46ec1415e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:53542/0 (socket says 172.21.15.94:53542) 2024-03-29T11:18:57.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46eb7fe640 1 -- 172.21.15.94:0/3351496772 learned_addr learned my addr 172.21.15.94:0/3351496772 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T11:18:57.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 10 client.?.objecter ms_handle_connect 0x7f46ec09ac40 2024-03-29T11:18:57.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:57.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 10 client.?.objecter ms_handle_connect 0x7f46ec140cc0 2024-03-29T11:18:57.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:57.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 10 client.?.objecter ms_handle_connect 0x7f46ec139420 2024-03-29T11:18:57.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:57.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 1 -- 172.21.15.94:0/3351496772 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1845970073 0 0) 0x7f46ec139890 con 0x7f46ec140cc0 2024-03-29T11:18:57.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:18:57.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:57.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:57.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 10 monclient(hunting): my global_id is 4142 2024-03-29T11:18:57.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 1 -- 172.21.15.94:0/3351496772 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46d8004eb0 con 0x7f46ec140cc0 2024-03-29T11:18:57.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 1 -- 172.21.15.94:0/3351496772 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2801159623 0 0) 0x7f46ec0afb40 con 0x7f46ec09ac40 2024-03-29T11:18:57.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:18:57.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:57.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:57.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 10 monclient(hunting): my global_id is 4144 2024-03-29T11:18:57.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 1 -- 172.21.15.94:0/3351496772 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46ec139890 con 0x7f46ec09ac40 2024-03-29T11:18:57.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 1 -- 172.21.15.94:0/3351496772 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3271711939 0 0) 0x7f46ec141cf0 con 0x7f46ec139420 2024-03-29T11:18:57.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:18:57.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:57.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:57.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 10 monclient(hunting): my global_id is 4140 2024-03-29T11:18:57.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 1 -- 172.21.15.94:0/3351496772 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46ec0afb40 con 0x7f46ec139420 2024-03-29T11:18:57.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 1 -- 172.21.15.94:0/3351496772 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3853008970 0 0) 0x7f46d8004eb0 con 0x7f46ec140cc0 2024-03-29T11:18:57.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 1 -- 172.21.15.94:0/3351496772 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46ec141cf0 con 0x7f46ec140cc0 2024-03-29T11:18:57.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 1 -- 172.21.15.94:0/3351496772 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f46e00018f0 con 0x7f46ec140cc0 2024-03-29T11:18:57.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:57.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:18:57.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 10 monclient(hunting): dump: 2024-03-29T11:18:57.722 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T11:18:57.722 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:57.722 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.722 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.722 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:57.722 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:57.722 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:57.722 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:57.722 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:57.722 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:57.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:57.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 1 -- 172.21.15.94:0/3351496772 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1786025672 0 0) 0x7f46ec139890 con 0x7f46ec09ac40 2024-03-29T11:18:57.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 1 -- 172.21.15.94:0/3351496772 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46d8004eb0 con 0x7f46ec09ac40 2024-03-29T11:18:57.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 1 -- 172.21.15.94:0/3351496772 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f46dc001ce0 con 0x7f46ec09ac40 2024-03-29T11:18:57.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:57.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:18:57.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 10 monclient(hunting): dump: 2024-03-29T11:18:57.723 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T11:18:57.723 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:57.723 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.723 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.723 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:57.723 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:57.723 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:57.723 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:57.723 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:57.723 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:57.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 1 -- 172.21.15.94:0/3351496772 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1719288911 0 0) 0x7f46ec0afb40 con 0x7f46ec139420 2024-03-29T11:18:57.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 1 -- 172.21.15.94:0/3351496772 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46ec139890 con 0x7f46ec139420 2024-03-29T11:18:57.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 1 -- 172.21.15.94:0/3351496772 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f46e40019e0 con 0x7f46ec139420 2024-03-29T11:18:57.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:57.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:57.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 10 monclient(hunting): dump: 2024-03-29T11:18:57.724 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T11:18:57.724 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:57.724 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.724 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.724 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:57.724 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:57.724 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:57.724 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:57.724 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:57.725 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:57.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 1 -- 172.21.15.94:0/3351496772 <== mon.2 v1:172.21.15.17:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 987225717 0 0) 0x7f46ec141cf0 con 0x7f46ec140cc0 2024-03-29T11:18:57.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 1 -- 172.21.15.94:0/3351496772 >> v1:172.21.15.94:6789/0 conn(0x7f46ec09ac40 legacy=0x7f46ec1415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:57.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 1 -- 172.21.15.94:0/3351496772 >> v1:172.21.15.17:6789/0 conn(0x7f46ec139420 legacy=0x7f46ec0affd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:57.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:18:57.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 1 monclient: found mon.c 2024-03-29T11:18:57.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:57.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:18:57.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 1 -- 172.21.15.94:0/3351496772 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46ec142b90 con 0x7f46ec140cc0 2024-03-29T11:18:57.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 10 monclient: _finish_auth 0 2024-03-29T11:18:57.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:18:57.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:57.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46f1e33640 5 monclient: authenticate success, global_id 4142 2024-03-29T11:18:57.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46f1e33640 10 monclient: _renew_subs 2024-03-29T11:18:57.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46f1e33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:18:57.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46f1e33640 1 -- 172.21.15.94:0/3351496772 --> v1:172.21.15.17:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46ec0b06e0 con 0x7f46ec140cc0 2024-03-29T11:18:57.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46f1e33640 10 client.4142.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:57.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46f1e33640 10 monclient: _renew_subs 2024-03-29T11:18:57.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46f1e33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:18:57.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46f1e33640 1 -- 172.21.15.94:0/3351496772 --> v1:172.21.15.17:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f46ec0b0b70 con 0x7f46ec140cc0 2024-03-29T11:18:57.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46f1e33640 1 librados: init done 2024-03-29T11:18:57.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 1 -- 172.21.15.94:0/3351496772 <== mon.2 v1:172.21.15.17:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46e0003610 con 0x7f46ec140cc0 2024-03-29T11:18:57.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:57.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 1 -- 172.21.15.94:0/3351496772 <== mon.2 v1:172.21.15.17:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f46e0003920 con 0x7f46ec140cc0 2024-03-29T11:18:57.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.731+0000 7f46d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:57.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.732+0000 7f46d3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:57.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.732+0000 7f46d3fff640 10 monclient: dump: 2024-03-29T11:18:57.727 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T11:18:57.727 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:57.727 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.727 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.727 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:57.727 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:57.727 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:57.727 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:57.727 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:57.727 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:57.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.732+0000 7f46e9ffb640 4 set_mon_vals no callback set 2024-03-29T11:18:57.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.732+0000 7f46d3fff640 1 -- 172.21.15.94:0/3351496772 <== mon.2 v1:172.21.15.17:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f46e0003e10 con 0x7f46ec140cc0 2024-03-29T11:18:57.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.732+0000 7f46d3fff640 1 -- 172.21.15.94:0/3351496772 <== mon.2 v1:172.21.15.17:6790/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 2513473704 0 0) 0x7f46e0005520 con 0x7f46ec140cc0 2024-03-29T11:18:57.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.732+0000 7f46d3fff640 10 client.4142.objecter ms_dispatch 0x7f46ec000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T11:18:57.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.732+0000 7f46d3fff640 3 client.4142.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T11:18:57.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.732+0000 7f46d3fff640 3 client.4142.objecter handle_osd_map decoding full epoch 6 2024-03-29T11:18:57.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.732+0000 7f46d3fff640 20 client.4142.objecter dump_active .. 0 homeless 2024-03-29T11:18:57.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.733+0000 7f46f1e33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:57.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.733+0000 7f46f1e33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:57.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.733+0000 7f46f1e33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:18:57.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.733+0000 7f46f1e33640 1 -- 172.21.15.94:0/3351496772 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46cc005080 con 0x7f46ec140cc0 2024-03-29T11:18:57.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.735+0000 7f46d3fff640 1 -- 172.21.15.94:0/3351496772 <== mon.2 v1:172.21.15.17:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f46e0005010 con 0x7f46ec140cc0 2024-03-29T11:18:57.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.735+0000 7f46d3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:57.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.735+0000 7f46d3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:57.729 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-03-29T11:18:57.750+0000 7fcdfee3c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:18:57.745 INFO:tasks.ceph.mgr.y.smithi017.stderr:2024-03-29T11:18:57.768+0000 7f6f3d22e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:18:57.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.807+0000 7f46f1e33640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f43c11bd-586c-4992-bb35-04023637a513", "id": 4}] 2024-03-29T11:18:57.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.807+0000 7f46f1e33640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f43c11bd-586c-4992-bb35-04023637a513", "id": 4}] 2024-03-29T11:18:57.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.807+0000 7f46f1e33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:18:57.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.807+0000 7f46f1e33640 1 -- 172.21.15.94:0/3351496772 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "osd new", "uuid": "f43c11bd-586c-4992-bb35-04023637a513", "id": 4} v 0) v1 -- 0x7f46cc0058b0 con 0x7f46ec140cc0 2024-03-29T11:18:57.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.810+0000 7f46d3fff640 1 -- 172.21.15.94:0/3351496772 <== mon.2 v1:172.21.15.17:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f43c11bd-586c-4992-bb35-04023637a513", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 2000627514 0 1697012583) 0x7f46e0005280 con 0x7f46ec140cc0 2024-03-29T11:18:57.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.810+0000 7f46d3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f43c11bd-586c-4992-bb35-04023637a513", "id": 4}] 2024-03-29T11:18:57.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.810+0000 7f46d3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:57.788 INFO:teuthology.orchestra.run.smithi094.stdout:4 2024-03-29T11:18:57.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.811+0000 7f46f1e33640 10 librados: watch_flush enter 2024-03-29T11:18:57.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.811+0000 7f46f1e33640 10 librados: watch_flush exit 2024-03-29T11:18:57.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.811+0000 7f46f1e33640 20 client.4142.objecter shutdown clearing up homeless session... 2024-03-29T11:18:57.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.811+0000 7f46f1e33640 10 client.4142.objecter successfully canceled tick 2024-03-29T11:18:57.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.811+0000 7f46f1e33640 10 monclient: shutdown 2024-03-29T11:18:57.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.811+0000 7f46f1e33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:57.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.811+0000 7f46f1e33640 1 -- 172.21.15.94:0/3351496772 >> v1:172.21.15.17:6790/0 conn(0x7f46ec140cc0 legacy=0x7f46ec138d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:57.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.811+0000 7f46f1e33640 1 -- 172.21.15.94:0/3351496772 shutdown_connections 2024-03-29T11:18:57.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.811+0000 7f46f1e33640 1 -- 172.21.15.94:0/3351496772 >> 172.21.15.94:0/3351496772 conn(0x7f46ec066b00 msgr2=0x7f46ec13a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:57.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.811+0000 7f46f1e33640 1 -- 172.21.15.94:0/3351496772 shutdown_connections 2024-03-29T11:18:57.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.811+0000 7f46f1e33640 1 -- 172.21.15.94:0/3351496772 wait complete. 2024-03-29T11:18:57.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.811+0000 7f46f1e33640 1 librados: shutdown 2024-03-29T11:18:57.794 INFO:tasks.ceph.mgr.y.smithi017.stderr:2024-03-29T11:18:57.817+0000 7f6f3d22e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:18:57.798 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster ceph osd new 27650969-0f63-4d75-b6c6-469e01ba4083 5 2024-03-29T11:18:57.800 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-03-29T11:18:57.824+0000 7fcdfee3c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:18:57.848 INFO:tasks.ceph.mgr.y.smithi017.stderr:2024-03-29T11:18:57.871+0000 7f6f3d22e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:18:57.849 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-03-29T11:18:57.872+0000 7fcdfee3c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:18:57.897 INFO:tasks.ceph.mgr.y.smithi017.stderr:2024-03-29T11:18:57.920+0000 7f6f3d22e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:18:57.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.920+0000 7f6e0a21e640 10 monclient: get_monmap_and_config 2024-03-29T11:18:57.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.920+0000 7f6e0a21e640 10 monclient: build_initial_monmap 2024-03-29T11:18:57.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.920+0000 7f6e0a21e640 10 monclient: monmap: 2024-03-29T11:18:57.897 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-03-29T11:18:57.897 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:57.897 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:57.921742+0000 2024-03-29T11:18:57.897 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:57.921742+0000 2024-03-29T11:18:57.898 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:57.898 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:57.898 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:18:57.898 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:18:57.898 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:18:57.898 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:57.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.921+0000 7f6e0a21e640 1 Processor -- start 2024-03-29T11:18:57.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.921+0000 7f6e0a21e640 1 -- start start 2024-03-29T11:18:57.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.921+0000 7f6e0a21e640 10 monclient: init 2024-03-29T11:18:57.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e0a21e640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:57.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e0a21e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:18:57.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e0a21e640 10 monclient(hunting): picked mon.noname-c con 0x7f6e040b1780 addr v1:172.21.15.94:6789/0 2024-03-29T11:18:57.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e0a21e640 10 monclient(hunting): picked mon.noname-b con 0x7f6e0413d7e0 addr v1:172.21.15.17:6790/0 2024-03-29T11:18:57.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e0a21e640 10 monclient(hunting): picked mon.noname-a con 0x7f6e04143be0 addr v1:172.21.15.17:6789/0 2024-03-29T11:18:57.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e0a21e640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e04070b30 con 0x7f6e04143be0 2024-03-29T11:18:57.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e0a21e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e04070da0 con 0x7f6e040b1780 2024-03-29T11:18:57.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e0a21e640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e0409adb0 con 0x7f6e0413d7e0 2024-03-29T11:18:57.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e0a21e640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:57.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e0a21e640 10 monclient(hunting): authenticate will time out at 823.083260s 2024-03-29T11:18:57.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e037fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f6e040b1780 0x7f6e0413d0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:53552/0 (socket says 172.21.15.94:53552) 2024-03-29T11:18:57.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e037fe640 1 -- 172.21.15.94:0/3661264352 learned_addr learned my addr 172.21.15.94:0/3661264352 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T11:18:57.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e027fc640 1 -- 172.21.15.94:0/3661264352 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 375777353 0 0) 0x7f6e04070da0 con 0x7f6e040b1780 2024-03-29T11:18:57.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e027fc640 10 monclient(hunting): _negotiate 2024-03-29T11:18:57.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e027fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:57.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:57.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e027fc640 10 monclient(hunting): my global_id is 4147 2024-03-29T11:18:57.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e027fc640 1 -- 172.21.15.94:0/3661264352 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dec003520 con 0x7f6e040b1780 2024-03-29T11:18:57.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e027fc640 1 -- 172.21.15.94:0/3661264352 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1402445200 0 0) 0x7f6e04070b30 con 0x7f6e04143be0 2024-03-29T11:18:57.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e027fc640 10 monclient(hunting): _negotiate 2024-03-29T11:18:57.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e027fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:57.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:57.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e027fc640 10 monclient(hunting): my global_id is 4143 2024-03-29T11:18:57.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e027fc640 1 -- 172.21.15.94:0/3661264352 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e04070da0 con 0x7f6e04143be0 2024-03-29T11:18:57.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e027fc640 1 -- 172.21.15.94:0/3661264352 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 215558296 0 0) 0x7f6e0409adb0 con 0x7f6e0413d7e0 2024-03-29T11:18:57.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e027fc640 10 monclient(hunting): _negotiate 2024-03-29T11:18:57.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e027fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:57.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:57.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e027fc640 10 monclient(hunting): my global_id is 4145 2024-03-29T11:18:57.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e027fc640 1 -- 172.21.15.94:0/3661264352 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e04070b30 con 0x7f6e0413d7e0 2024-03-29T11:18:57.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e027fc640 1 -- 172.21.15.94:0/3661264352 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2823003722 0 0) 0x7f6dec003520 con 0x7f6e040b1780 2024-03-29T11:18:57.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e027fc640 1 -- 172.21.15.94:0/3661264352 >> v1:172.21.15.17:6790/0 conn(0x7f6e0413d7e0 legacy=0x7f6e041434d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:57.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e027fc640 1 -- 172.21.15.94:0/3661264352 >> v1:172.21.15.17:6789/0 conn(0x7f6e04143be0 legacy=0x7f6e0414b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:57.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e027fc640 10 monclient: _finish_hunting 0 2024-03-29T11:18:57.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e027fc640 1 monclient: found mon.noname-c 2024-03-29T11:18:57.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:57.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e027fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6789/0 2024-03-29T11:18:57.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e027fc640 1 -- 172.21.15.94:0/3661264352 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e04144070 con 0x7f6e040b1780 2024-03-29T11:18:57.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e027fc640 10 monclient: _finish_auth 0 2024-03-29T11:18:57.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e027fc640 10 monclient: _check_auth_tickets 2024-03-29T11:18:57.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:57.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e0a21e640 5 monclient: authenticate success, global_id 4147 2024-03-29T11:18:57.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e027fc640 1 -- 172.21.15.94:0/3661264352 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f6df4001690 con 0x7f6e040b1780 2024-03-29T11:18:57.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:57.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e027fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:18:57.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e027fc640 10 monclient: dump: 2024-03-29T11:18:57.910 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T11:18:57.910 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:57.910 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.910 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.910 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:57.910 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:57.911 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:57.911 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:57.911 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:57.911 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:57.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e0a21e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:57.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e027fc640 1 -- 172.21.15.94:0/3661264352 <== mon.1 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6df4001870 con 0x7f6e040b1780 2024-03-29T11:18:57.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:57.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e027fc640 1 -- 172.21.15.94:0/3661264352 <== mon.1 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f6df4001c80 con 0x7f6e040b1780 2024-03-29T11:18:57.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e0a21e640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:57.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e0a21e640 4 set_mon_vals no callback set 2024-03-29T11:18:57.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:57.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:18:57.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.922+0000 7f6e027fc640 10 monclient: dump: 2024-03-29T11:18:57.911 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T11:18:57.911 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:57.912 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.912 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.912 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:57.912 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:57.912 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:57.912 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:57.912 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:57.912 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:57.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.923+0000 7f6e0a21e640 10 monclient: shutdown 2024-03-29T11:18:57.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.923+0000 7f6e0a21e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:57.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.923+0000 7f6e0a21e640 1 -- 172.21.15.94:0/3661264352 >> v1:172.21.15.94:6789/0 conn(0x7f6e040b1780 legacy=0x7f6e0413d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:57.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.923+0000 7f6e0a21e640 1 -- 172.21.15.94:0/3661264352 shutdown_connections 2024-03-29T11:18:57.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.923+0000 7f6e0a21e640 1 -- 172.21.15.94:0/3661264352 >> 172.21.15.94:0/3661264352 conn(0x7f6e040b58f0 msgr2=0x7f6e040b5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:57.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.923+0000 7f6e0a21e640 1 -- 172.21.15.94:0/3661264352 shutdown_connections 2024-03-29T11:18:57.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.923+0000 7f6e0a21e640 1 -- 172.21.15.94:0/3661264352 wait complete. 2024-03-29T11:18:57.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.924+0000 7f6e0a21e640 10 monclient: build_initial_monmap 2024-03-29T11:18:57.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.924+0000 7f6e0a21e640 10 monclient: monmap: 2024-03-29T11:18:57.913 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-03-29T11:18:57.913 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:57.913 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-03-29T11:18:57.913 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-03-29T11:18:57.913 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:57.913 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:57.913 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:18:57.913 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:18:57.913 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:18:57.913 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:57.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.924+0000 7f6e0a21e640 1 librados: starting msgr at 2024-03-29T11:18:57.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.924+0000 7f6e0a21e640 1 librados: starting objecter 2024-03-29T11:18:57.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.924+0000 7f6e0a21e640 1 Processor -- start 2024-03-29T11:18:57.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.924+0000 7f6e0a21e640 1 -- start start 2024-03-29T11:18:57.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.924+0000 7f6e0a21e640 1 librados: setting wanted keys 2024-03-29T11:18:57.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.924+0000 7f6e0a21e640 1 librados: calling monclient init 2024-03-29T11:18:57.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.924+0000 7f6e0a21e640 10 monclient: init 2024-03-29T11:18:57.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.925+0000 7f6e0a21e640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:57.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.925+0000 7f6e0a21e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:18:57.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.925+0000 7f6e0a21e640 10 monclient(hunting): picked mon.noname-c con 0x7f6e040b1780 addr v1:172.21.15.17:6790/0 2024-03-29T11:18:57.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.925+0000 7f6e0a21e640 10 monclient(hunting): picked mon.noname-a con 0x7f6e0413d7e0 addr v1:172.21.15.17:6789/0 2024-03-29T11:18:57.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.925+0000 7f6e0a21e640 10 monclient(hunting): picked mon.noname-b con 0x7f6e04143be0 addr v1:172.21.15.94:6789/0 2024-03-29T11:18:57.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.925+0000 7f6e0a21e640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e0413a200 con 0x7f6e0413d7e0 2024-03-29T11:18:57.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.925+0000 7f6e0a21e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e0413a4e0 con 0x7f6e04143be0 2024-03-29T11:18:57.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.925+0000 7f6e0a21e640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e041db2e0 con 0x7f6e040b1780 2024-03-29T11:18:57.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.925+0000 7f6e0a21e640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:57.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.925+0000 7f6e0a21e640 10 monclient(hunting): authenticate will time out at 823.086356s 2024-03-29T11:18:57.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.925+0000 7f6e03fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f6e04143be0 0x7f6e041e28e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:53554/0 (socket says 172.21.15.94:53554) 2024-03-29T11:18:57.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.925+0000 7f6e03fff640 1 -- 172.21.15.94:0/2720616287 learned_addr learned my addr 172.21.15.94:0/2720616287 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T11:18:57.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.925+0000 7f6debfff640 10 client.?.objecter ms_handle_connect 0x7f6e04143be0 2024-03-29T11:18:57.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.925+0000 7f6debfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:57.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.925+0000 7f6debfff640 10 client.?.objecter ms_handle_connect 0x7f6e0413d7e0 2024-03-29T11:18:57.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.925+0000 7f6debfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:57.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.925+0000 7f6debfff640 10 client.?.objecter ms_handle_connect 0x7f6e040b1780 2024-03-29T11:18:57.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.925+0000 7f6debfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:57.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.925+0000 7f6debfff640 1 -- 172.21.15.94:0/2720616287 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1481715593 0 0) 0x7f6e0413a4e0 con 0x7f6e04143be0 2024-03-29T11:18:57.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.925+0000 7f6debfff640 10 monclient(hunting): _negotiate 2024-03-29T11:18:57.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.925+0000 7f6debfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:57.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.925+0000 7f6debfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:57.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.925+0000 7f6debfff640 10 monclient(hunting): my global_id is 4150 2024-03-29T11:18:57.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.925+0000 7f6debfff640 1 -- 172.21.15.94:0/2720616287 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6df0004eb0 con 0x7f6e04143be0 2024-03-29T11:18:57.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.925+0000 7f6debfff640 1 -- 172.21.15.94:0/2720616287 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2585292419 0 0) 0x7f6e041db2e0 con 0x7f6e040b1780 2024-03-29T11:18:57.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.925+0000 7f6debfff640 10 monclient(hunting): _negotiate 2024-03-29T11:18:57.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.925+0000 7f6debfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:57.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.925+0000 7f6debfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:57.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.925+0000 7f6debfff640 10 monclient(hunting): my global_id is 4148 2024-03-29T11:18:57.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.925+0000 7f6debfff640 1 -- 172.21.15.94:0/2720616287 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e0413a4e0 con 0x7f6e040b1780 2024-03-29T11:18:57.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.925+0000 7f6debfff640 1 -- 172.21.15.94:0/2720616287 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 876464880 0 0) 0x7f6e0413a200 con 0x7f6e0413d7e0 2024-03-29T11:18:57.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.925+0000 7f6debfff640 10 monclient(hunting): _negotiate 2024-03-29T11:18:57.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.925+0000 7f6debfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:57.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.925+0000 7f6debfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:57.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.925+0000 7f6debfff640 10 monclient(hunting): my global_id is 4146 2024-03-29T11:18:57.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.925+0000 7f6debfff640 1 -- 172.21.15.94:0/2720616287 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e041db2e0 con 0x7f6e0413d7e0 2024-03-29T11:18:57.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.925+0000 7f6debfff640 1 -- 172.21.15.94:0/2720616287 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3934793627 0 0) 0x7f6e0413a4e0 con 0x7f6e040b1780 2024-03-29T11:18:57.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.925+0000 7f6debfff640 1 -- 172.21.15.94:0/2720616287 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e0413a200 con 0x7f6e040b1780 2024-03-29T11:18:57.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.925+0000 7f6debfff640 1 -- 172.21.15.94:0/2720616287 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f6df4001b80 con 0x7f6e040b1780 2024-03-29T11:18:57.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.925+0000 7f6debfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:57.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.925+0000 7f6debfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:18:57.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.925+0000 7f6debfff640 10 monclient(hunting): dump: 2024-03-29T11:18:57.917 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T11:18:57.917 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:57.917 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.917 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.917 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:57.917 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:57.917 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:57.917 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:57.917 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:57.917 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:57.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.925+0000 7f6debfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:57.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.925+0000 7f6debfff640 1 -- 172.21.15.94:0/2720616287 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 557400367 0 0) 0x7f6e041db2e0 con 0x7f6e0413d7e0 2024-03-29T11:18:57.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.925+0000 7f6debfff640 1 -- 172.21.15.94:0/2720616287 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e0413a4e0 con 0x7f6e0413d7e0 2024-03-29T11:18:57.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.925+0000 7f6debfff640 1 -- 172.21.15.94:0/2720616287 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4081841113 0 0) 0x7f6df0004eb0 con 0x7f6e04143be0 2024-03-29T11:18:57.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.926+0000 7f6debfff640 1 -- 172.21.15.94:0/2720616287 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e041db2e0 con 0x7f6e04143be0 2024-03-29T11:18:57.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.926+0000 7f6debfff640 1 -- 172.21.15.94:0/2720616287 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f6df8001770 con 0x7f6e0413d7e0 2024-03-29T11:18:57.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.926+0000 7f6debfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:57.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.926+0000 7f6debfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:57.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.926+0000 7f6debfff640 10 monclient(hunting): dump: 2024-03-29T11:18:57.918 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T11:18:57.918 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:57.918 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.918 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.918 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:57.919 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:57.919 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:57.919 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:57.919 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:57.919 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:57.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.926+0000 7f6debfff640 1 -- 172.21.15.94:0/2720616287 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f6dfc001c10 con 0x7f6e04143be0 2024-03-29T11:18:57.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.926+0000 7f6debfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:57.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.926+0000 7f6debfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:18:57.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.926+0000 7f6debfff640 10 monclient(hunting): dump: 2024-03-29T11:18:57.919 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T11:18:57.919 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:57.919 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.919 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.919 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:57.919 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:57.920 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:57.920 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:57.920 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:57.920 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:57.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.926+0000 7f6debfff640 1 -- 172.21.15.94:0/2720616287 <== mon.2 v1:172.21.15.17:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4223594908 0 0) 0x7f6e0413a200 con 0x7f6e040b1780 2024-03-29T11:18:57.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.926+0000 7f6debfff640 1 -- 172.21.15.94:0/2720616287 >> v1:172.21.15.94:6789/0 conn(0x7f6e04143be0 legacy=0x7f6e041e28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:57.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.926+0000 7f6debfff640 1 -- 172.21.15.94:0/2720616287 >> v1:172.21.15.17:6789/0 conn(0x7f6e0413d7e0 legacy=0x7f6e04140340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:57.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.926+0000 7f6debfff640 10 monclient: _finish_hunting 0 2024-03-29T11:18:57.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.926+0000 7f6debfff640 1 monclient: found mon.c 2024-03-29T11:18:57.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.926+0000 7f6debfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:57.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.926+0000 7f6debfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:18:57.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.926+0000 7f6debfff640 1 -- 172.21.15.94:0/2720616287 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e041daa00 con 0x7f6e040b1780 2024-03-29T11:18:57.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.926+0000 7f6debfff640 10 monclient: _finish_auth 0 2024-03-29T11:18:57.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.926+0000 7f6debfff640 10 monclient: _check_auth_tickets 2024-03-29T11:18:57.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.926+0000 7f6debfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:57.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.926+0000 7f6e0a21e640 5 monclient: authenticate success, global_id 4148 2024-03-29T11:18:57.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.926+0000 7f6e0a21e640 10 monclient: _renew_subs 2024-03-29T11:18:57.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.926+0000 7f6e0a21e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:18:57.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.926+0000 7f6e0a21e640 1 -- 172.21.15.94:0/2720616287 --> v1:172.21.15.17:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e0406c980 con 0x7f6e040b1780 2024-03-29T11:18:57.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.926+0000 7f6e0a21e640 10 client.4148.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:57.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.926+0000 7f6e0a21e640 10 monclient: _renew_subs 2024-03-29T11:18:57.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.926+0000 7f6e0a21e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:18:57.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.926+0000 7f6e0a21e640 1 -- 172.21.15.94:0/2720616287 --> v1:172.21.15.17:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e041d78b0 con 0x7f6e040b1780 2024-03-29T11:18:57.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.926+0000 7f6e0a21e640 1 librados: init done 2024-03-29T11:18:57.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.926+0000 7f6debfff640 1 -- 172.21.15.94:0/2720616287 <== mon.2 v1:172.21.15.17:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6df4004bf0 con 0x7f6e040b1780 2024-03-29T11:18:57.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.926+0000 7f6debfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:57.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.926+0000 7f6debfff640 1 -- 172.21.15.94:0/2720616287 <== mon.2 v1:172.21.15.17:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f6df4004f00 con 0x7f6e040b1780 2024-03-29T11:18:57.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.926+0000 7f6debfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:57.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.926+0000 7f6e01ffb640 4 set_mon_vals no callback set 2024-03-29T11:18:57.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.926+0000 7f6debfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:57.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.926+0000 7f6debfff640 10 monclient: dump: 2024-03-29T11:18:57.921 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T11:18:57.922 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:57.922 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.922 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:57.922 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:57.922 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:57.922 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:57.922 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:57.922 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:57.922 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:57.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.926+0000 7f6debfff640 1 -- 172.21.15.94:0/2720616287 <== mon.2 v1:172.21.15.17:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6df40053f0 con 0x7f6e040b1780 2024-03-29T11:18:57.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.926+0000 7f6debfff640 1 -- 172.21.15.94:0/2720616287 <== mon.2 v1:172.21.15.17:6790/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 1027973158 0 0) 0x7f6df4006a60 con 0x7f6e040b1780 2024-03-29T11:18:57.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.926+0000 7f6debfff640 10 client.4148.objecter ms_dispatch 0x7f6e04000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T11:18:57.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.926+0000 7f6debfff640 3 client.4148.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T11:18:57.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.926+0000 7f6debfff640 3 client.4148.objecter handle_osd_map decoding full epoch 7 2024-03-29T11:18:57.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.926+0000 7f6debfff640 20 client.4148.objecter dump_active .. 0 homeless 2024-03-29T11:18:57.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.927+0000 7f6e0a21e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:57.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.927+0000 7f6e0a21e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:57.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.927+0000 7f6e0a21e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:18:57.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.927+0000 7f6e0a21e640 1 -- 172.21.15.94:0/2720616287 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6de4005080 con 0x7f6e040b1780 2024-03-29T11:18:57.923 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-03-29T11:18:57.928+0000 7fcdfee3c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:18:57.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.929+0000 7f6debfff640 1 -- 172.21.15.94:0/2720616287 <== mon.2 v1:172.21.15.17:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6df4006590 con 0x7f6e040b1780 2024-03-29T11:18:57.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.929+0000 7f6debfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:57.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:57.929+0000 7f6debfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:57.954 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-03-29T11:18:57.977+0000 7fcdfee3c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:18:57.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.001+0000 7f6e0a21e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "27650969-0f63-4d75-b6c6-469e01ba4083", "id": 5}] 2024-03-29T11:18:57.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.001+0000 7f6e0a21e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "27650969-0f63-4d75-b6c6-469e01ba4083", "id": 5}] 2024-03-29T11:18:57.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.001+0000 7f6e0a21e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:18:57.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.001+0000 7f6e0a21e640 1 -- 172.21.15.94:0/2720616287 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "osd new", "uuid": "27650969-0f63-4d75-b6c6-469e01ba4083", "id": 5} v 0) v1 -- 0x7f6de40058b0 con 0x7f6e040b1780 2024-03-29T11:18:57.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.004+0000 7f6debfff640 1 -- 172.21.15.94:0/2720616287 <== mon.2 v1:172.21.15.17:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "27650969-0f63-4d75-b6c6-469e01ba4083", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 3118163294 0 1988412176) 0x7f6df4005740 con 0x7f6e040b1780 2024-03-29T11:18:57.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.004+0000 7f6debfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "27650969-0f63-4d75-b6c6-469e01ba4083", "id": 5}] 2024-03-29T11:18:57.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.004+0000 7f6debfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:57.981 INFO:teuthology.orchestra.run.smithi094.stdout:5 2024-03-29T11:18:57.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.005+0000 7f6e0a21e640 10 librados: watch_flush enter 2024-03-29T11:18:57.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.005+0000 7f6e0a21e640 10 librados: watch_flush exit 2024-03-29T11:18:57.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.005+0000 7f6e0a21e640 20 client.4148.objecter shutdown clearing up homeless session... 2024-03-29T11:18:57.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.005+0000 7f6e0a21e640 10 client.4148.objecter successfully canceled tick 2024-03-29T11:18:57.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.005+0000 7f6e0a21e640 10 monclient: shutdown 2024-03-29T11:18:57.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.005+0000 7f6e0a21e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:57.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.005+0000 7f6e0a21e640 1 -- 172.21.15.94:0/2720616287 >> v1:172.21.15.17:6790/0 conn(0x7f6e040b1780 legacy=0x7f6e041d6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:57.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.005+0000 7f6e0a21e640 1 -- 172.21.15.94:0/2720616287 shutdown_connections 2024-03-29T11:18:57.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.005+0000 7f6e0a21e640 1 -- 172.21.15.94:0/2720616287 >> 172.21.15.94:0/2720616287 conn(0x7f6e040b58f0 msgr2=0x7f6e04069bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:57.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.005+0000 7f6e0a21e640 1 -- 172.21.15.94:0/2720616287 shutdown_connections 2024-03-29T11:18:57.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.005+0000 7f6e0a21e640 1 -- 172.21.15.94:0/2720616287 wait complete. 2024-03-29T11:18:57.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.005+0000 7f6e0a21e640 1 librados: shutdown 2024-03-29T11:18:57.993 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster ceph osd new ffc86661-1c24-44aa-a9aa-a45fe8816e2a 6 2024-03-29T11:18:58.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.115+0000 7fc0152a8640 10 monclient: get_monmap_and_config 2024-03-29T11:18:58.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.115+0000 7fc0152a8640 10 monclient: build_initial_monmap 2024-03-29T11:18:58.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.115+0000 7fc0152a8640 10 monclient: monmap: 2024-03-29T11:18:58.093 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-03-29T11:18:58.093 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:58.093 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:58.116823+0000 2024-03-29T11:18:58.093 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:58.116823+0000 2024-03-29T11:18:58.093 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:58.093 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:58.093 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:18:58.093 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:18:58.094 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:18:58.094 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:58.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.116+0000 7fc0152a8640 1 Processor -- start 2024-03-29T11:18:58.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.116+0000 7fc0152a8640 1 -- start start 2024-03-29T11:18:58.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.116+0000 7fc0152a8640 10 monclient: init 2024-03-29T11:18:58.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc0152a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:58.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc0152a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:18:58.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc0152a8640 10 monclient(hunting): picked mon.noname-b con 0x7fc01009ac40 addr v1:172.21.15.17:6790/0 2024-03-29T11:18:58.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc0152a8640 10 monclient(hunting): picked mon.noname-c con 0x7fc01009c870 addr v1:172.21.15.94:6789/0 2024-03-29T11:18:58.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc0152a8640 10 monclient(hunting): picked mon.noname-a con 0x7fc0100b4360 addr v1:172.21.15.17:6789/0 2024-03-29T11:18:58.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc0152a8640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc01009dec0 con 0x7fc0100b4360 2024-03-29T11:18:58.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc0152a8640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc01009da30 con 0x7fc01009c870 2024-03-29T11:18:58.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc0152a8640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0100b39c0 con 0x7fc01009ac40 2024-03-29T11:18:58.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc0152a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:58.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc0152a8640 10 monclient(hunting): authenticate will time out at 823.278157s 2024-03-29T11:18:58.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc00e7fc640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fc01009c870 0x7fc01013f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:53560/0 (socket says 172.21.15.94:53560) 2024-03-29T11:18:58.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc00e7fc640 1 -- 172.21.15.94:0/2636297909 learned_addr learned my addr 172.21.15.94:0/2636297909 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T11:18:58.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc00dffb640 1 -- 172.21.15.94:0/2636297909 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2558837163 0 0) 0x7fc01009da30 con 0x7fc01009c870 2024-03-29T11:18:58.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc00dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:18:58.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc00dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:58.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc00dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:58.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc00dffb640 10 monclient(hunting): my global_id is 4153 2024-03-29T11:18:58.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc00dffb640 1 -- 172.21.15.94:0/2636297909 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbff8003520 con 0x7fc01009c870 2024-03-29T11:18:58.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc00dffb640 1 -- 172.21.15.94:0/2636297909 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4070196147 0 0) 0x7fc008001490 con 0x7fc0100b4360 2024-03-29T11:18:58.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc00dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:18:58.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc00dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:58.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc00dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:58.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc00dffb640 10 monclient(hunting): my global_id is 4149 2024-03-29T11:18:58.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc00dffb640 1 -- 172.21.15.94:0/2636297909 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc01009da30 con 0x7fc0100b4360 2024-03-29T11:18:58.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc00dffb640 1 -- 172.21.15.94:0/2636297909 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2977091266 0 0) 0x7fc0100b39c0 con 0x7fc01009ac40 2024-03-29T11:18:58.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc00dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:18:58.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc00dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:58.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc00dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:58.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc00dffb640 10 monclient(hunting): my global_id is 4151 2024-03-29T11:18:58.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc00dffb640 1 -- 172.21.15.94:0/2636297909 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc008001490 con 0x7fc01009ac40 2024-03-29T11:18:58.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc00dffb640 1 -- 172.21.15.94:0/2636297909 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 993273926 0 0) 0x7fbff8003520 con 0x7fc01009c870 2024-03-29T11:18:58.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc00dffb640 1 -- 172.21.15.94:0/2636297909 >> v1:172.21.15.17:6790/0 conn(0x7fc01009ac40 legacy=0x7fc0100b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:58.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc00dffb640 1 -- 172.21.15.94:0/2636297909 >> v1:172.21.15.17:6789/0 conn(0x7fc0100b4360 legacy=0x7fc01014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:58.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc00dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:18:58.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc00dffb640 1 monclient: found mon.noname-c 2024-03-29T11:18:58.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc00dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:58.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc00dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6789/0 2024-03-29T11:18:58.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc00dffb640 1 -- 172.21.15.94:0/2636297909 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0100b5160 con 0x7fc01009c870 2024-03-29T11:18:58.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc00dffb640 10 monclient: _finish_auth 0 2024-03-29T11:18:58.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc00dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:18:58.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc00dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:58.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc0152a8640 5 monclient: authenticate success, global_id 4153 2024-03-29T11:18:58.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc00dffb640 1 -- 172.21.15.94:0/2636297909 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fc000001690 con 0x7fc01009c870 2024-03-29T11:18:58.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc00dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:58.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc00dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:18:58.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc00dffb640 10 monclient: dump: 2024-03-29T11:18:58.106 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T11:18:58.106 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:58.106 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:58.106 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:58.106 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:58.106 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:58.106 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:58.106 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:58.106 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:58.106 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:58.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc0152a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:58.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc00dffb640 1 -- 172.21.15.94:0/2636297909 <== mon.1 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc000001870 con 0x7fc01009c870 2024-03-29T11:18:58.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc00dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:58.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc0152a8640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:58.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc00dffb640 1 -- 172.21.15.94:0/2636297909 <== mon.1 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fc000001c80 con 0x7fc01009c870 2024-03-29T11:18:58.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc0152a8640 4 set_mon_vals no callback set 2024-03-29T11:18:58.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc00dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:58.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc00dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:18:58.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.117+0000 7fc00dffb640 10 monclient: dump: 2024-03-29T11:18:58.107 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T11:18:58.107 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:58.107 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:58.107 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:58.107 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:58.107 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:58.107 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:58.107 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:58.107 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:58.108 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:58.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.118+0000 7fc0152a8640 10 monclient: shutdown 2024-03-29T11:18:58.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.118+0000 7fc0152a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:58.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.118+0000 7fc0152a8640 1 -- 172.21.15.94:0/2636297909 >> v1:172.21.15.94:6789/0 conn(0x7fc01009c870 legacy=0x7fc01013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:58.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.118+0000 7fc0152a8640 1 -- 172.21.15.94:0/2636297909 shutdown_connections 2024-03-29T11:18:58.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.118+0000 7fc0152a8640 1 -- 172.21.15.94:0/2636297909 >> 172.21.15.94:0/2636297909 conn(0x7fc010066b00 msgr2=0x7fc010066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:58.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.118+0000 7fc0152a8640 1 -- 172.21.15.94:0/2636297909 shutdown_connections 2024-03-29T11:18:58.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.118+0000 7fc0152a8640 1 -- 172.21.15.94:0/2636297909 wait complete. 2024-03-29T11:18:58.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.119+0000 7fc0152a8640 10 monclient: build_initial_monmap 2024-03-29T11:18:58.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.119+0000 7fc0152a8640 10 monclient: monmap: 2024-03-29T11:18:58.108 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-03-29T11:18:58.108 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:58.108 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-03-29T11:18:58.108 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-03-29T11:18:58.109 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:58.109 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:58.109 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:18:58.109 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:18:58.109 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:18:58.109 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:58.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.119+0000 7fc0152a8640 1 librados: starting msgr at 2024-03-29T11:18:58.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.119+0000 7fc0152a8640 1 librados: starting objecter 2024-03-29T11:18:58.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.119+0000 7fc0152a8640 1 Processor -- start 2024-03-29T11:18:58.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.119+0000 7fc0152a8640 1 -- start start 2024-03-29T11:18:58.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.119+0000 7fc0152a8640 1 librados: setting wanted keys 2024-03-29T11:18:58.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.119+0000 7fc0152a8640 1 librados: calling monclient init 2024-03-29T11:18:58.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.119+0000 7fc0152a8640 10 monclient: init 2024-03-29T11:18:58.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fc0152a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:58.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fc0152a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:18:58.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fc0152a8640 10 monclient(hunting): picked mon.noname-b con 0x7fc01009ac40 addr v1:172.21.15.94:6789/0 2024-03-29T11:18:58.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fc0152a8640 10 monclient(hunting): picked mon.noname-c con 0x7fc0101d7ab0 addr v1:172.21.15.17:6790/0 2024-03-29T11:18:58.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fc0152a8640 10 monclient(hunting): picked mon.noname-a con 0x7fc010140fb0 addr v1:172.21.15.17:6789/0 2024-03-29T11:18:58.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fc0152a8640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc010142480 con 0x7fc010140fb0 2024-03-29T11:18:58.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fc0152a8640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0101db090 con 0x7fc01009ac40 2024-03-29T11:18:58.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fc0152a8640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0101dad80 con 0x7fc0101d7ab0 2024-03-29T11:18:58.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fc0152a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:58.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fc0152a8640 10 monclient(hunting): authenticate will time out at 823.281297s 2024-03-29T11:18:58.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fc00effd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fc01009ac40 0x7fc0101d73a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:53570/0 (socket says 172.21.15.94:53570) 2024-03-29T11:18:58.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fc00effd640 1 -- 172.21.15.94:0/3576035785 learned_addr learned my addr 172.21.15.94:0/3576035785 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T11:18:58.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fbff77fe640 10 client.?.objecter ms_handle_connect 0x7fc0101d7ab0 2024-03-29T11:18:58.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fbff77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:58.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fbff77fe640 10 client.?.objecter ms_handle_connect 0x7fc010140fb0 2024-03-29T11:18:58.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fbff77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:58.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fbff77fe640 10 client.?.objecter ms_handle_connect 0x7fc01009ac40 2024-03-29T11:18:58.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fbff77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:58.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fbff77fe640 1 -- 172.21.15.94:0/3576035785 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4133036685 0 0) 0x7fc0101dad80 con 0x7fc0101d7ab0 2024-03-29T11:18:58.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fbff77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:18:58.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fbff77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:58.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fbff77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:58.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fbff77fe640 10 monclient(hunting): my global_id is 4154 2024-03-29T11:18:58.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fbff77fe640 1 -- 172.21.15.94:0/3576035785 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbffc004eb0 con 0x7fc0101d7ab0 2024-03-29T11:18:58.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fbff77fe640 1 -- 172.21.15.94:0/3576035785 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3888879457 0 0) 0x7fc0101db090 con 0x7fc01009ac40 2024-03-29T11:18:58.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fbff77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:18:58.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fbff77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:58.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fbff77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:58.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fbff77fe640 10 monclient(hunting): my global_id is 4156 2024-03-29T11:18:58.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fbff77fe640 1 -- 172.21.15.94:0/3576035785 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0101dad80 con 0x7fc01009ac40 2024-03-29T11:18:58.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fbff77fe640 1 -- 172.21.15.94:0/3576035785 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4216154035 0 0) 0x7fc010142480 con 0x7fc010140fb0 2024-03-29T11:18:58.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fbff77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:18:58.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fbff77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:58.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fbff77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:58.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fbff77fe640 10 monclient(hunting): my global_id is 4152 2024-03-29T11:18:58.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fbff77fe640 1 -- 172.21.15.94:0/3576035785 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0101db090 con 0x7fc010140fb0 2024-03-29T11:18:58.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fbff77fe640 1 -- 172.21.15.94:0/3576035785 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 85306114 0 0) 0x7fbffc004eb0 con 0x7fc0101d7ab0 2024-03-29T11:18:58.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fbff77fe640 1 -- 172.21.15.94:0/3576035785 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc010142480 con 0x7fc0101d7ab0 2024-03-29T11:18:58.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fbff77fe640 1 -- 172.21.15.94:0/3576035785 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fc000001b80 con 0x7fc0101d7ab0 2024-03-29T11:18:58.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fbff77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:58.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fbff77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:18:58.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fbff77fe640 10 monclient(hunting): dump: 2024-03-29T11:18:58.112 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T11:18:58.112 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:58.112 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:58.112 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:58.113 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:58.113 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:58.113 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:58.113 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:58.113 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:58.113 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:58.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fbff77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:58.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fbff77fe640 1 -- 172.21.15.94:0/3576035785 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 448458350 0 0) 0x7fc0101dad80 con 0x7fc01009ac40 2024-03-29T11:18:58.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fbff77fe640 1 -- 172.21.15.94:0/3576035785 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbffc004eb0 con 0x7fc01009ac40 2024-03-29T11:18:58.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fbff77fe640 1 -- 172.21.15.94:0/3576035785 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3113328235 0 0) 0x7fc0101db090 con 0x7fc010140fb0 2024-03-29T11:18:58.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fbff77fe640 1 -- 172.21.15.94:0/3576035785 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc0101dad80 con 0x7fc010140fb0 2024-03-29T11:18:58.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fbff77fe640 1 -- 172.21.15.94:0/3576035785 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fc004001970 con 0x7fc01009ac40 2024-03-29T11:18:58.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fbff77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:58.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fbff77fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:18:58.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fbff77fe640 10 monclient(hunting): dump: 2024-03-29T11:18:58.114 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T11:18:58.114 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:58.114 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:58.114 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:58.114 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:58.114 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:58.114 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:58.114 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:58.114 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:58.114 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:58.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fbff77fe640 1 -- 172.21.15.94:0/3576035785 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fc008001a40 con 0x7fc010140fb0 2024-03-29T11:18:58.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fbff77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:58.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fbff77fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:58.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fbff77fe640 10 monclient(hunting): dump: 2024-03-29T11:18:58.115 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T11:18:58.115 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:58.115 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:58.115 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:58.115 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:58.115 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:58.115 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:58.115 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:58.115 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:58.115 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:58.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.120+0000 7fbff77fe640 1 -- 172.21.15.94:0/3576035785 <== mon.2 v1:172.21.15.17:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3180011043 0 0) 0x7fc010142480 con 0x7fc0101d7ab0 2024-03-29T11:18:58.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.121+0000 7fbff77fe640 1 -- 172.21.15.94:0/3576035785 >> v1:172.21.15.94:6789/0 conn(0x7fc01009ac40 legacy=0x7fc0101d73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:58.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.121+0000 7fbff77fe640 1 -- 172.21.15.94:0/3576035785 >> v1:172.21.15.17:6789/0 conn(0x7fc010140fb0 legacy=0x7fc0101e2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:58.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.121+0000 7fbff77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:18:58.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.121+0000 7fbff77fe640 1 monclient: found mon.c 2024-03-29T11:18:58.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.121+0000 7fbff77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:58.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.121+0000 7fbff77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:18:58.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.121+0000 7fbff77fe640 1 -- 172.21.15.94:0/3576035785 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0100b7db0 con 0x7fc0101d7ab0 2024-03-29T11:18:58.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.121+0000 7fbff77fe640 10 monclient: _finish_auth 0 2024-03-29T11:18:58.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.121+0000 7fbff77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:18:58.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.121+0000 7fbff77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:58.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.121+0000 7fc0152a8640 5 monclient: authenticate success, global_id 4154 2024-03-29T11:18:58.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.121+0000 7fc0152a8640 10 monclient: _renew_subs 2024-03-29T11:18:58.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.121+0000 7fc0152a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:18:58.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.121+0000 7fc0152a8640 1 -- 172.21.15.94:0/3576035785 --> v1:172.21.15.17:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0101db5b0 con 0x7fc0101d7ab0 2024-03-29T11:18:58.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.121+0000 7fc0152a8640 10 client.4154.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:58.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.121+0000 7fc0152a8640 10 monclient: _renew_subs 2024-03-29T11:18:58.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.121+0000 7fc0152a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:18:58.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.121+0000 7fc0152a8640 1 -- 172.21.15.94:0/3576035785 --> v1:172.21.15.17:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc010142d80 con 0x7fc0101d7ab0 2024-03-29T11:18:58.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.121+0000 7fc0152a8640 1 librados: init done 2024-03-29T11:18:58.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.121+0000 7fbff77fe640 1 -- 172.21.15.94:0/3576035785 <== mon.2 v1:172.21.15.17:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc000004bf0 con 0x7fc0101d7ab0 2024-03-29T11:18:58.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.121+0000 7fbff77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:58.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.121+0000 7fbff77fe640 1 -- 172.21.15.94:0/3576035785 <== mon.2 v1:172.21.15.17:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fc000004f00 con 0x7fc0101d7ab0 2024-03-29T11:18:58.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.121+0000 7fbff77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:58.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.121+0000 7fbff77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:58.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.121+0000 7fc00d7fa640 4 set_mon_vals no callback set 2024-03-29T11:18:58.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.121+0000 7fbff77fe640 10 monclient: dump: 2024-03-29T11:18:58.117 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T11:18:58.117 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:58.117 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:58.117 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:58.117 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:58.117 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:58.117 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:58.117 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:58.118 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:58.118 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:58.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.121+0000 7fbff77fe640 1 -- 172.21.15.94:0/3576035785 <== mon.2 v1:172.21.15.17:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fc0000053f0 con 0x7fc0101d7ab0 2024-03-29T11:18:58.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.121+0000 7fbff77fe640 1 -- 172.21.15.94:0/3576035785 <== mon.2 v1:172.21.15.17:6790/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 2079415508 0 0) 0x7fc000006b20 con 0x7fc0101d7ab0 2024-03-29T11:18:58.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.121+0000 7fbff77fe640 10 client.4154.objecter ms_dispatch 0x7fc010000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T11:18:58.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.121+0000 7fbff77fe640 3 client.4154.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T11:18:58.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.121+0000 7fbff77fe640 3 client.4154.objecter handle_osd_map decoding full epoch 8 2024-03-29T11:18:58.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.121+0000 7fbff77fe640 20 client.4154.objecter dump_active .. 0 homeless 2024-03-29T11:18:58.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.121+0000 7fc0152a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:58.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.121+0000 7fc0152a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:58.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.121+0000 7fc0152a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:18:58.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.121+0000 7fc0152a8640 1 -- 172.21.15.94:0/3576035785 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbff0005080 con 0x7fc0101d7ab0 2024-03-29T11:18:58.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.124+0000 7fbff77fe640 1 -- 172.21.15.94:0/3576035785 <== mon.2 v1:172.21.15.17:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fc000006530 con 0x7fc0101d7ab0 2024-03-29T11:18:58.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.124+0000 7fbff77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:58.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.124+0000 7fbff77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:58.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.196+0000 7fc0152a8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ffc86661-1c24-44aa-a9aa-a45fe8816e2a", "id": 6}] 2024-03-29T11:18:58.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.196+0000 7fc0152a8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ffc86661-1c24-44aa-a9aa-a45fe8816e2a", "id": 6}] 2024-03-29T11:18:58.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.196+0000 7fc0152a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:18:58.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.196+0000 7fc0152a8640 1 -- 172.21.15.94:0/3576035785 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "osd new", "uuid": "ffc86661-1c24-44aa-a9aa-a45fe8816e2a", "id": 6} v 0) v1 -- 0x7fbff00058b0 con 0x7fc0101d7ab0 2024-03-29T11:18:58.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.199+0000 7fbff77fe640 1 -- 172.21.15.94:0/3576035785 <== mon.2 v1:172.21.15.17:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ffc86661-1c24-44aa-a9aa-a45fe8816e2a", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 1543910062 0 1113811849) 0x7fc000006820 con 0x7fc0101d7ab0 2024-03-29T11:18:58.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.199+0000 7fbff77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ffc86661-1c24-44aa-a9aa-a45fe8816e2a", "id": 6}] 2024-03-29T11:18:58.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.199+0000 7fbff77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:58.176 INFO:teuthology.orchestra.run.smithi094.stdout:6 2024-03-29T11:18:58.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.200+0000 7fc0152a8640 10 librados: watch_flush enter 2024-03-29T11:18:58.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.200+0000 7fc0152a8640 10 librados: watch_flush exit 2024-03-29T11:18:58.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.200+0000 7fc0152a8640 20 client.4154.objecter shutdown clearing up homeless session... 2024-03-29T11:18:58.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.200+0000 7fc0152a8640 10 client.4154.objecter successfully canceled tick 2024-03-29T11:18:58.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.200+0000 7fc0152a8640 10 monclient: shutdown 2024-03-29T11:18:58.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.200+0000 7fc0152a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:58.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.200+0000 7fc0152a8640 1 -- 172.21.15.94:0/3576035785 >> v1:172.21.15.17:6790/0 conn(0x7fc0101d7ab0 legacy=0x7fc0100b1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:58.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.200+0000 7fc0152a8640 1 -- 172.21.15.94:0/3576035785 shutdown_connections 2024-03-29T11:18:58.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.200+0000 7fc0152a8640 1 -- 172.21.15.94:0/3576035785 >> 172.21.15.94:0/3576035785 conn(0x7fc010066b00 msgr2=0x7fc010066ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:58.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.200+0000 7fc0152a8640 1 -- 172.21.15.94:0/3576035785 shutdown_connections 2024-03-29T11:18:58.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.200+0000 7fc0152a8640 1 -- 172.21.15.94:0/3576035785 wait complete. 2024-03-29T11:18:58.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.200+0000 7fc0152a8640 1 librados: shutdown 2024-03-29T11:18:58.187 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster ceph osd new 379f15ab-e903-42a2-82cd-d222df0861e2 7 2024-03-29T11:18:58.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.310+0000 7f54e0589640 10 monclient: get_monmap_and_config 2024-03-29T11:18:58.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.310+0000 7f54e0589640 10 monclient: build_initial_monmap 2024-03-29T11:18:58.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.310+0000 7f54e0589640 10 monclient: monmap: 2024-03-29T11:18:58.288 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-03-29T11:18:58.288 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:58.288 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:58.311794+0000 2024-03-29T11:18:58.288 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:58.311794+0000 2024-03-29T11:18:58.288 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:58.288 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:58.288 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:18:58.288 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:18:58.288 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:18:58.288 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:58.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.311+0000 7f54e0589640 1 Processor -- start 2024-03-29T11:18:58.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.311+0000 7f54e0589640 1 -- start start 2024-03-29T11:18:58.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.311+0000 7f54e0589640 10 monclient: init 2024-03-29T11:18:58.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.312+0000 7f54e0589640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:58.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.312+0000 7f54e0589640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:18:58.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.312+0000 7f54e0589640 10 monclient(hunting): picked mon.noname-a con 0x7f54d809ac40 addr v1:172.21.15.17:6789/0 2024-03-29T11:18:58.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.312+0000 7f54e0589640 10 monclient(hunting): picked mon.noname-b con 0x7f54d809c870 addr v1:172.21.15.17:6790/0 2024-03-29T11:18:58.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.312+0000 7f54e0589640 10 monclient(hunting): picked mon.noname-c con 0x7f54d80b4360 addr v1:172.21.15.94:6789/0 2024-03-29T11:18:58.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.312+0000 7f54e0589640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54d809dec0 con 0x7f54d809ac40 2024-03-29T11:18:58.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.312+0000 7f54e0589640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54d809da30 con 0x7f54d80b4360 2024-03-29T11:18:58.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.312+0000 7f54e0589640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54d80b39c0 con 0x7f54d809c870 2024-03-29T11:18:58.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.312+0000 7f54e0589640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:58.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.312+0000 7f54e0589640 10 monclient(hunting): authenticate will time out at 823.473397s 2024-03-29T11:18:58.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.312+0000 7f54deb26640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f54d80b4360 0x7f54d814d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:53584/0 (socket says 172.21.15.94:53584) 2024-03-29T11:18:58.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.312+0000 7f54deb26640 1 -- 172.21.15.94:0/4289054203 learned_addr learned my addr 172.21.15.94:0/4289054203 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T11:18:58.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.312+0000 7f54dd323640 1 -- 172.21.15.94:0/4289054203 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1364185964 0 0) 0x7f54d809da30 con 0x7f54d80b4360 2024-03-29T11:18:58.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.312+0000 7f54dd323640 10 monclient(hunting): _negotiate 2024-03-29T11:18:58.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.312+0000 7f54dd323640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:58.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.312+0000 7f54dd323640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:58.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.312+0000 7f54dd323640 10 monclient(hunting): my global_id is 4159 2024-03-29T11:18:58.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.312+0000 7f54dd323640 1 -- 172.21.15.94:0/4289054203 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54c4003520 con 0x7f54d80b4360 2024-03-29T11:18:58.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.312+0000 7f54dd323640 1 -- 172.21.15.94:0/4289054203 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 940582645 0 0) 0x7f54c8001110 con 0x7f54d809ac40 2024-03-29T11:18:58.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.312+0000 7f54dd323640 10 monclient(hunting): _negotiate 2024-03-29T11:18:58.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.312+0000 7f54dd323640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:58.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.312+0000 7f54dd323640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:58.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.312+0000 7f54dd323640 10 monclient(hunting): my global_id is 4155 2024-03-29T11:18:58.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.312+0000 7f54dd323640 1 -- 172.21.15.94:0/4289054203 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54d809da30 con 0x7f54d809ac40 2024-03-29T11:18:58.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.312+0000 7f54dd323640 1 -- 172.21.15.94:0/4289054203 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3521302423 0 0) 0x7f54d80b39c0 con 0x7f54d809c870 2024-03-29T11:18:58.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.312+0000 7f54dd323640 10 monclient(hunting): _negotiate 2024-03-29T11:18:58.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.312+0000 7f54dd323640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:58.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.312+0000 7f54dd323640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:58.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.312+0000 7f54dd323640 10 monclient(hunting): my global_id is 4157 2024-03-29T11:18:58.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.312+0000 7f54dd323640 1 -- 172.21.15.94:0/4289054203 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54c8001110 con 0x7f54d809c870 2024-03-29T11:18:58.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.312+0000 7f54dd323640 1 -- 172.21.15.94:0/4289054203 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4196515730 0 0) 0x7f54c4003520 con 0x7f54d80b4360 2024-03-29T11:18:58.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.312+0000 7f54dd323640 1 -- 172.21.15.94:0/4289054203 >> v1:172.21.15.17:6790/0 conn(0x7f54d809c870 legacy=0x7f54d813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:58.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.312+0000 7f54dd323640 1 -- 172.21.15.94:0/4289054203 >> v1:172.21.15.17:6789/0 conn(0x7f54d809ac40 legacy=0x7f54d80b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:58.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.312+0000 7f54dd323640 10 monclient: _finish_hunting 0 2024-03-29T11:18:58.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.312+0000 7f54dd323640 1 monclient: found mon.noname-c 2024-03-29T11:18:58.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.312+0000 7f54dd323640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:58.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.312+0000 7f54dd323640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6789/0 2024-03-29T11:18:58.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.312+0000 7f54dd323640 1 -- 172.21.15.94:0/4289054203 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d80b5160 con 0x7f54d80b4360 2024-03-29T11:18:58.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.312+0000 7f54dd323640 10 monclient: _finish_auth 0 2024-03-29T11:18:58.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.312+0000 7f54dd323640 10 monclient: _check_auth_tickets 2024-03-29T11:18:58.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.312+0000 7f54dd323640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:58.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.312+0000 7f54dd323640 1 -- 172.21.15.94:0/4289054203 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f54d0001a10 con 0x7f54d80b4360 2024-03-29T11:18:58.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.312+0000 7f54e0589640 5 monclient: authenticate success, global_id 4159 2024-03-29T11:18:58.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.312+0000 7f54e0589640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:58.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.312+0000 7f54dd323640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:58.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.313+0000 7f54dd323640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:18:58.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.313+0000 7f54dd323640 10 monclient: dump: 2024-03-29T11:18:58.301 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T11:18:58.301 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:58.301 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:58.301 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:58.301 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:58.301 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:58.302 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:58.302 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:58.302 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:58.302 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:58.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.313+0000 7f54e0589640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:58.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.313+0000 7f54dd323640 1 -- 172.21.15.94:0/4289054203 <== mon.1 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54d0001bf0 con 0x7f54d80b4360 2024-03-29T11:18:58.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.313+0000 7f54dd323640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:58.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.313+0000 7f54e0589640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:58.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.313+0000 7f54e0589640 4 set_mon_vals no callback set 2024-03-29T11:18:58.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.313+0000 7f54dd323640 1 -- 172.21.15.94:0/4289054203 <== mon.1 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f54d0002000 con 0x7f54d80b4360 2024-03-29T11:18:58.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.313+0000 7f54dd323640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:58.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.313+0000 7f54dd323640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:18:58.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.313+0000 7f54dd323640 10 monclient: dump: 2024-03-29T11:18:58.302 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T11:18:58.302 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:58.303 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:58.303 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:58.303 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:58.303 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:58.303 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:58.303 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:58.303 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:58.303 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:58.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.313+0000 7f54e0589640 10 monclient: shutdown 2024-03-29T11:18:58.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.313+0000 7f54e0589640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:58.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.313+0000 7f54e0589640 1 -- 172.21.15.94:0/4289054203 >> v1:172.21.15.94:6789/0 conn(0x7f54d80b4360 legacy=0x7f54d814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:58.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.313+0000 7f54e0589640 1 -- 172.21.15.94:0/4289054203 shutdown_connections 2024-03-29T11:18:58.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.313+0000 7f54e0589640 1 -- 172.21.15.94:0/4289054203 >> 172.21.15.94:0/4289054203 conn(0x7f54d8066b00 msgr2=0x7f54d8066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:58.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.313+0000 7f54e0589640 1 -- 172.21.15.94:0/4289054203 shutdown_connections 2024-03-29T11:18:58.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.313+0000 7f54e0589640 1 -- 172.21.15.94:0/4289054203 wait complete. 2024-03-29T11:18:58.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.314+0000 7f54e0589640 10 monclient: build_initial_monmap 2024-03-29T11:18:58.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.314+0000 7f54e0589640 10 monclient: monmap: 2024-03-29T11:18:58.304 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-03-29T11:18:58.304 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:58.304 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-03-29T11:18:58.304 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-03-29T11:18:58.304 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:58.304 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:58.304 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:18:58.304 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:18:58.304 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:18:58.304 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:58.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.314+0000 7f54e0589640 1 librados: starting msgr at 2024-03-29T11:18:58.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.314+0000 7f54e0589640 1 librados: starting objecter 2024-03-29T11:18:58.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.314+0000 7f54e0589640 1 Processor -- start 2024-03-29T11:18:58.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.314+0000 7f54e0589640 1 -- start start 2024-03-29T11:18:58.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.314+0000 7f54e0589640 1 librados: setting wanted keys 2024-03-29T11:18:58.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.314+0000 7f54e0589640 1 librados: calling monclient init 2024-03-29T11:18:58.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.314+0000 7f54e0589640 10 monclient: init 2024-03-29T11:18:58.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.315+0000 7f54e0589640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:58.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.315+0000 7f54e0589640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:18:58.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.315+0000 7f54e0589640 10 monclient(hunting): picked mon.noname-a con 0x7f54d809ac40 addr v1:172.21.15.17:6789/0 2024-03-29T11:18:58.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.315+0000 7f54e0589640 10 monclient(hunting): picked mon.noname-c con 0x7f54d81436c0 addr v1:172.21.15.17:6790/0 2024-03-29T11:18:58.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.315+0000 7f54e0589640 10 monclient(hunting): picked mon.noname-b con 0x7f54d81e0000 addr v1:172.21.15.94:6789/0 2024-03-29T11:18:58.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.315+0000 7f54e0589640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54d8142ff0 con 0x7f54d809ac40 2024-03-29T11:18:58.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.315+0000 7f54e0589640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54d8145210 con 0x7f54d81e0000 2024-03-29T11:18:58.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.315+0000 7f54e0589640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54d8144210 con 0x7f54d81436c0 2024-03-29T11:18:58.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.315+0000 7f54e0589640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:58.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.315+0000 7f54e0589640 10 monclient(hunting): authenticate will time out at 823.476414s 2024-03-29T11:18:58.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.315+0000 7f54deb26640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f54d81e0000 0x7f54d81e23f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:53600/0 (socket says 172.21.15.94:53600) 2024-03-29T11:18:58.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.315+0000 7f54deb26640 1 -- 172.21.15.94:0/3657079794 learned_addr learned my addr 172.21.15.94:0/3657079794 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T11:18:58.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.315+0000 7f54ceffd640 10 client.?.objecter ms_handle_connect 0x7f54d81e0000 2024-03-29T11:18:58.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.315+0000 7f54ceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:58.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.315+0000 7f54ceffd640 10 client.?.objecter ms_handle_connect 0x7f54d809ac40 2024-03-29T11:18:58.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.315+0000 7f54ceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:58.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.315+0000 7f54ceffd640 10 client.?.objecter ms_handle_connect 0x7f54d81436c0 2024-03-29T11:18:58.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.315+0000 7f54ceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:58.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.315+0000 7f54ceffd640 1 -- 172.21.15.94:0/3657079794 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 146528642 0 0) 0x7f54d8145210 con 0x7f54d81e0000 2024-03-29T11:18:58.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.315+0000 7f54ceffd640 10 monclient(hunting): _negotiate 2024-03-29T11:18:58.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.315+0000 7f54ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:58.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.315+0000 7f54ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:58.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.315+0000 7f54ceffd640 10 monclient(hunting): my global_id is 4162 2024-03-29T11:18:58.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.315+0000 7f54ceffd640 1 -- 172.21.15.94:0/3657079794 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54bc004eb0 con 0x7f54d81e0000 2024-03-29T11:18:58.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.315+0000 7f54ceffd640 1 -- 172.21.15.94:0/3657079794 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3187839053 0 0) 0x7f54d8142ff0 con 0x7f54d809ac40 2024-03-29T11:18:58.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.315+0000 7f54ceffd640 10 monclient(hunting): _negotiate 2024-03-29T11:18:58.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.315+0000 7f54ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:58.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.315+0000 7f54ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:58.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.315+0000 7f54ceffd640 10 monclient(hunting): my global_id is 4158 2024-03-29T11:18:58.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.315+0000 7f54ceffd640 1 -- 172.21.15.94:0/3657079794 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54d8145210 con 0x7f54d809ac40 2024-03-29T11:18:58.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.315+0000 7f54ceffd640 1 -- 172.21.15.94:0/3657079794 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4184462351 0 0) 0x7f54d8144210 con 0x7f54d81436c0 2024-03-29T11:18:58.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.315+0000 7f54ceffd640 10 monclient(hunting): _negotiate 2024-03-29T11:18:58.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.315+0000 7f54ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:58.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.315+0000 7f54ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:58.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.315+0000 7f54ceffd640 10 monclient(hunting): my global_id is 4160 2024-03-29T11:18:58.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.315+0000 7f54ceffd640 1 -- 172.21.15.94:0/3657079794 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54d8142ff0 con 0x7f54d81436c0 2024-03-29T11:18:58.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.315+0000 7f54ceffd640 1 -- 172.21.15.94:0/3657079794 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4120544429 0 0) 0x7f54d8145210 con 0x7f54d809ac40 2024-03-29T11:18:58.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.315+0000 7f54ceffd640 1 -- 172.21.15.94:0/3657079794 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54d8144210 con 0x7f54d809ac40 2024-03-29T11:18:58.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.315+0000 7f54ceffd640 1 -- 172.21.15.94:0/3657079794 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f54c8001920 con 0x7f54d809ac40 2024-03-29T11:18:58.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.315+0000 7f54ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:58.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54ceffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:18:58.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54ceffd640 10 monclient(hunting): dump: 2024-03-29T11:18:58.308 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T11:18:58.308 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:58.308 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:58.308 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:58.308 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:58.308 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:58.308 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:58.308 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:58.308 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:58.308 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:58.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54ceffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:58.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54ceffd640 1 -- 172.21.15.94:0/3657079794 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2311751979 0 0) 0x7f54d8142ff0 con 0x7f54d81436c0 2024-03-29T11:18:58.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54ceffd640 1 -- 172.21.15.94:0/3657079794 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54d8145210 con 0x7f54d81436c0 2024-03-29T11:18:58.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54ceffd640 1 -- 172.21.15.94:0/3657079794 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1299429193 0 0) 0x7f54bc004eb0 con 0x7f54d81e0000 2024-03-29T11:18:58.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54ceffd640 1 -- 172.21.15.94:0/3657079794 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54d8142ff0 con 0x7f54d81e0000 2024-03-29T11:18:58.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54ceffd640 1 -- 172.21.15.94:0/3657079794 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f54d4001830 con 0x7f54d81436c0 2024-03-29T11:18:58.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:58.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54ceffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:58.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54ceffd640 10 monclient(hunting): dump: 2024-03-29T11:18:58.309 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T11:18:58.309 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:58.309 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:58.309 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:58.309 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:58.309 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:58.309 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:58.309 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:58.309 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:58.309 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:58.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54ceffd640 1 -- 172.21.15.94:0/3657079794 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f54d0001f00 con 0x7f54d81e0000 2024-03-29T11:18:58.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:58.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54ceffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:18:58.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54ceffd640 10 monclient(hunting): dump: 2024-03-29T11:18:58.310 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T11:18:58.310 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:58.310 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:58.310 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:58.310 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:58.310 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:58.310 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:58.310 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:58.310 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:58.310 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:58.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54ceffd640 1 -- 172.21.15.94:0/3657079794 <== mon.0 v1:172.21.15.17:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1477140952 0 0) 0x7f54d8144210 con 0x7f54d809ac40 2024-03-29T11:18:58.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54ceffd640 1 -- 172.21.15.94:0/3657079794 >> v1:172.21.15.17:6790/0 conn(0x7f54d81436c0 legacy=0x7f54d8143b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:58.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54ceffd640 1 -- 172.21.15.94:0/3657079794 >> v1:172.21.15.94:6789/0 conn(0x7f54d81e0000 legacy=0x7f54d81e23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:58.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54ceffd640 10 monclient: _finish_hunting 0 2024-03-29T11:18:58.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54ceffd640 1 monclient: found mon.a 2024-03-29T11:18:58.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:58.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54ceffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:18:58.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54ceffd640 1 -- 172.21.15.94:0/3657079794 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d8142b80 con 0x7f54d809ac40 2024-03-29T11:18:58.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54ceffd640 10 monclient: _finish_auth 0 2024-03-29T11:18:58.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54ceffd640 10 monclient: _check_auth_tickets 2024-03-29T11:18:58.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:58.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54e0589640 5 monclient: authenticate success, global_id 4158 2024-03-29T11:18:58.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54e0589640 10 monclient: _renew_subs 2024-03-29T11:18:58.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54e0589640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:18:58.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54e0589640 1 -- 172.21.15.94:0/3657079794 --> v1:172.21.15.17:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54d8146390 con 0x7f54d809ac40 2024-03-29T11:18:58.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54e0589640 10 client.4158.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:58.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54e0589640 10 monclient: _renew_subs 2024-03-29T11:18:58.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54e0589640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:18:58.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54e0589640 1 -- 172.21.15.94:0/3657079794 --> v1:172.21.15.17:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f54d81e3cf0 con 0x7f54d809ac40 2024-03-29T11:18:58.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54e0589640 1 librados: init done 2024-03-29T11:18:58.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54ceffd640 1 -- 172.21.15.94:0/3657079794 <== mon.0 v1:172.21.15.17:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54c80035f0 con 0x7f54d809ac40 2024-03-29T11:18:58.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54ceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:58.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54ceffd640 1 -- 172.21.15.94:0/3657079794 <== mon.0 v1:172.21.15.17:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f54c8003900 con 0x7f54d809ac40 2024-03-29T11:18:58.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:58.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54ceffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:58.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54dcb22640 4 set_mon_vals no callback set 2024-03-29T11:18:58.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54ceffd640 10 monclient: dump: 2024-03-29T11:18:58.312 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T11:18:58.312 INFO:teuthology.orchestra.run.smithi094.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:58.312 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:58.313 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:58.313 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:58.313 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T11:18:58.313 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:58.313 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:58.313 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:58.313 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:18:58.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54ceffd640 1 -- 172.21.15.94:0/3657079794 <== mon.0 v1:172.21.15.17:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f54c8003df0 con 0x7f54d809ac40 2024-03-29T11:18:58.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54ceffd640 1 -- 172.21.15.94:0/3657079794 <== mon.0 v1:172.21.15.17:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 2568775366 0 0) 0x7f54c8005690 con 0x7f54d809ac40 2024-03-29T11:18:58.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54ceffd640 10 client.4158.objecter ms_dispatch 0x7f54d8000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T11:18:58.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54ceffd640 3 client.4158.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T11:18:58.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54ceffd640 3 client.4158.objecter handle_osd_map decoding full epoch 9 2024-03-29T11:18:58.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.316+0000 7f54ceffd640 20 client.4158.objecter dump_active .. 0 homeless 2024-03-29T11:18:58.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.317+0000 7f54e0589640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:58.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.317+0000 7f54e0589640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:58.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.317+0000 7f54e0589640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:18:58.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.317+0000 7f54e0589640 1 -- 172.21.15.94:0/3657079794 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54c0005080 con 0x7f54d809ac40 2024-03-29T11:18:58.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.319+0000 7f54ceffd640 1 -- 172.21.15.94:0/3657079794 <== mon.0 v1:172.21.15.17:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f54c8005140 con 0x7f54d809ac40 2024-03-29T11:18:58.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.319+0000 7f54ceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:58.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.319+0000 7f54ceffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:58.314 INFO:tasks.ceph.mgr.y.smithi017.stderr:2024-03-29T11:18:58.320+0000 7f6f3d22e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:18:58.355 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-03-29T11:18:58.379+0000 7fcdfee3c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:18:58.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.391+0000 7f54e0589640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "379f15ab-e903-42a2-82cd-d222df0861e2", "id": 7}] 2024-03-29T11:18:58.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.391+0000 7f54e0589640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "379f15ab-e903-42a2-82cd-d222df0861e2", "id": 7}] 2024-03-29T11:18:58.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.391+0000 7f54e0589640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:18:58.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.391+0000 7f54e0589640 1 -- 172.21.15.94:0/3657079794 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "osd new", "uuid": "379f15ab-e903-42a2-82cd-d222df0861e2", "id": 7} v 0) v1 -- 0x7f54c00058b0 con 0x7f54d809ac40 2024-03-29T11:18:58.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.393+0000 7f54ceffd640 1 -- 172.21.15.94:0/3657079794 <== mon.0 v1:172.21.15.17:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "379f15ab-e903-42a2-82cd-d222df0861e2", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 2351882453 0 1371665406) 0x7f54c80053b0 con 0x7f54d809ac40 2024-03-29T11:18:58.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.393+0000 7f54ceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "379f15ab-e903-42a2-82cd-d222df0861e2", "id": 7}] 2024-03-29T11:18:58.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.393+0000 7f54ceffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:58.370 INFO:teuthology.orchestra.run.smithi094.stdout:7 2024-03-29T11:18:58.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.394+0000 7f54e0589640 10 librados: watch_flush enter 2024-03-29T11:18:58.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.394+0000 7f54e0589640 10 librados: watch_flush exit 2024-03-29T11:18:58.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.394+0000 7f54e0589640 20 client.4158.objecter shutdown clearing up homeless session... 2024-03-29T11:18:58.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.394+0000 7f54e0589640 10 client.4158.objecter successfully canceled tick 2024-03-29T11:18:58.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.394+0000 7f54e0589640 10 monclient: shutdown 2024-03-29T11:18:58.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.394+0000 7f54e0589640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:58.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.394+0000 7f54e0589640 1 -- 172.21.15.94:0/3657079794 >> v1:172.21.15.17:6789/0 conn(0x7f54d809ac40 legacy=0x7f54d8145c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:58.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.394+0000 7f54e0589640 1 -- 172.21.15.94:0/3657079794 shutdown_connections 2024-03-29T11:18:58.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.394+0000 7f54e0589640 1 -- 172.21.15.94:0/3657079794 >> 172.21.15.94:0/3657079794 conn(0x7f54d8066b00 msgr2=0x7f54d813a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:58.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.394+0000 7f54e0589640 1 -- 172.21.15.94:0/3657079794 shutdown_connections 2024-03-29T11:18:58.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.394+0000 7f54e0589640 1 -- 172.21.15.94:0/3657079794 wait complete. 2024-03-29T11:18:58.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T11:18:58.394+0000 7f54e0589640 1 librados: shutdown 2024-03-29T11:18:58.380 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T11:18:58.381 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T11:18:58.383 INFO:tasks.ceph.osd.0:Started 2024-03-29T11:18:58.383 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T11:18:58.383 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T11:18:58.385 INFO:tasks.ceph.osd.1:Started 2024-03-29T11:18:58.385 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T11:18:58.385 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T11:18:58.387 INFO:tasks.ceph.osd.2:Started 2024-03-29T11:18:58.387 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T11:18:58.387 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T11:18:58.389 INFO:tasks.ceph.osd.3:Started 2024-03-29T11:18:58.389 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T11:18:58.389 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T11:18:58.391 INFO:tasks.ceph.osd.4:Started 2024-03-29T11:18:58.391 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T11:18:58.392 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T11:18:58.393 INFO:tasks.ceph.osd.5:Started 2024-03-29T11:18:58.394 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T11:18:58.394 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T11:18:58.396 INFO:tasks.ceph.osd.6:Started 2024-03-29T11:18:58.396 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T11:18:58.396 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T11:18:58.398 INFO:tasks.ceph.osd.7:Started 2024-03-29T11:18:58.398 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:58.470 INFO:tasks.ceph.mgr.y.smithi017.stderr:2024-03-29T11:18:58.493+0000 7f6f3d22e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:18:58.515 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-03-29T11:18:58.538+0000 7fcdfee3c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:18:58.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.540+0000 7ff291b1c640 10 monclient: get_monmap_and_config 2024-03-29T11:18:58.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.540+0000 7ff291b1c640 10 monclient: build_initial_monmap 2024-03-29T11:18:58.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.540+0000 7ff291b1c640 10 monclient: monmap: 2024-03-29T11:18:58.517 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-03-29T11:18:58.517 INFO:teuthology.orchestra.run.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:58.517 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-03-29T11:18:58.541184+0000 2024-03-29T11:18:58.517 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-03-29T11:18:58.541184+0000 2024-03-29T11:18:58.517 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:58.517 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-03-29T11:18:58.517 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:18:58.517 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:18:58.517 INFO:teuthology.orchestra.run.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:18:58.517 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-03-29T11:18:58.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.540+0000 7ff291b1c640 1 Processor -- start 2024-03-29T11:18:58.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.541+0000 7ff291b1c640 1 -- start start 2024-03-29T11:18:58.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.541+0000 7ff291b1c640 10 monclient: init 2024-03-29T11:18:58.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.541+0000 7ff291b1c640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:58.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.541+0000 7ff291b1c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:18:58.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.541+0000 7ff291b1c640 10 monclient(hunting): picked mon.noname-b con 0x7ff28c0b8240 addr v1:172.21.15.17:6790/0 2024-03-29T11:18:58.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.541+0000 7ff291b1c640 10 monclient(hunting): picked mon.noname-a con 0x7ff28c0b1f60 addr v1:172.21.15.17:6789/0 2024-03-29T11:18:58.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.541+0000 7ff291b1c640 10 monclient(hunting): picked mon.noname-c con 0x7ff28c09c8c0 addr v1:172.21.15.94:6789/0 2024-03-29T11:18:58.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.541+0000 7ff28b7fe640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7ff28c0b8240 0x7ff28c13f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:42344/0 (socket says 172.21.15.17:42344) 2024-03-29T11:18:58.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.541+0000 7ff282ffd640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7ff28c0b1f60 0x7ff28c09c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:58854/0 (socket says 172.21.15.17:58854) 2024-03-29T11:18:58.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff28b7fe640 1 -- 172.21.15.17:0/3626919185 learned_addr learned my addr 172.21.15.17:0/3626919185 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:18:58.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff291b1c640 1 -- 172.21.15.17:0/3626919185 --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff28c09bd80 con 0x7ff28c0b1f60 2024-03-29T11:18:58.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff291b1c640 1 -- 172.21.15.17:0/3626919185 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff28c09b960 con 0x7ff28c09c8c0 2024-03-29T11:18:58.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff291b1c640 1 -- 172.21.15.17:0/3626919185 --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff28c0b2ec0 con 0x7ff28c0b8240 2024-03-29T11:18:58.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff291b1c640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:58.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff291b1c640 10 monclient(hunting): authenticate will time out at 844.746970s 2024-03-29T11:18:58.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff28affd640 1 -- 172.21.15.17:0/3626919185 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3339587821 0 0) 0x7ff28c0b2ec0 con 0x7ff28c0b8240 2024-03-29T11:18:58.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff28affd640 10 monclient(hunting): _negotiate 2024-03-29T11:18:58.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff28affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:58.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff28affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:58.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff28affd640 10 monclient(hunting): my global_id is 4187 2024-03-29T11:18:58.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff28affd640 1 -- 172.21.15.17:0/3626919185 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff278003520 con 0x7ff28c0b8240 2024-03-29T11:18:58.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff28affd640 1 -- 172.21.15.17:0/3626919185 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3421919973 0 0) 0x7ff28c09bd80 con 0x7ff28c0b1f60 2024-03-29T11:18:58.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff28affd640 10 monclient(hunting): _negotiate 2024-03-29T11:18:58.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff28affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:58.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff28affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:58.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff28affd640 10 monclient(hunting): my global_id is 4185 2024-03-29T11:18:58.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff28affd640 1 -- 172.21.15.17:0/3626919185 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff28c0b2ec0 con 0x7ff28c0b1f60 2024-03-29T11:18:58.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff28affd640 1 -- 172.21.15.17:0/3626919185 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1902578852 0 0) 0x7ff28c09b960 con 0x7ff28c09c8c0 2024-03-29T11:18:58.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff28affd640 10 monclient(hunting): _negotiate 2024-03-29T11:18:58.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff28affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:58.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff28affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:58.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff28affd640 10 monclient(hunting): my global_id is 4189 2024-03-29T11:18:58.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff28affd640 1 -- 172.21.15.17:0/3626919185 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff28c09bd80 con 0x7ff28c09c8c0 2024-03-29T11:18:58.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff28affd640 1 -- 172.21.15.17:0/3626919185 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1556192367 0 0) 0x7ff28c0b2ec0 con 0x7ff28c0b1f60 2024-03-29T11:18:58.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff28affd640 1 -- 172.21.15.17:0/3626919185 >> v1:172.21.15.17:6790/0 conn(0x7ff28c0b8240 legacy=0x7ff28c13f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:58.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff28affd640 1 -- 172.21.15.17:0/3626919185 >> v1:172.21.15.94:6789/0 conn(0x7ff28c09c8c0 legacy=0x7ff28c14d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:58.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff28affd640 10 monclient: _finish_hunting 0 2024-03-29T11:18:58.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff28affd640 1 monclient: found mon.noname-a 2024-03-29T11:18:58.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff28affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:58.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff28affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.17:6789/0 2024-03-29T11:18:58.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff28affd640 1 -- 172.21.15.17:0/3626919185 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff28c13fb60 con 0x7ff28c0b1f60 2024-03-29T11:18:58.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff28affd640 10 monclient: _finish_auth 0 2024-03-29T11:18:58.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff28affd640 10 monclient: _check_auth_tickets 2024-03-29T11:18:58.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff28affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:58.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff28affd640 1 -- 172.21.15.17:0/3626919185 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7ff274001520 con 0x7ff28c0b1f60 2024-03-29T11:18:58.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff28affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:58.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff28affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:18:58.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff28affd640 10 monclient: dump: 2024-03-29T11:18:58.528 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-03-29T11:18:58.528 INFO:teuthology.orchestra.run.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:58.529 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:58.529 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:58.529 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:58.529 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-03-29T11:18:58.529 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:58.529 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:58.529 INFO:teuthology.orchestra.run.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:58.529 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-03-29T11:18:58.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff291b1c640 5 monclient: authenticate success, global_id 4185 2024-03-29T11:18:58.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff291b1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:58.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff28affd640 1 -- 172.21.15.17:0/3626919185 <== mon.0 v1:172.21.15.17:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff274001680 con 0x7ff28c0b1f60 2024-03-29T11:18:58.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff28affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:58.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff28affd640 1 -- 172.21.15.17:0/3626919185 <== mon.0 v1:172.21.15.17:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7ff274001a90 con 0x7ff28c0b1f60 2024-03-29T11:18:58.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff291b1c640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:58.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff291b1c640 4 set_mon_vals no callback set 2024-03-29T11:18:58.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff28affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:58.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff28affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:58.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.542+0000 7ff28affd640 10 monclient: dump: 2024-03-29T11:18:58.530 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-03-29T11:18:58.530 INFO:teuthology.orchestra.run.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:58.530 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:58.530 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:58.530 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:58.530 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-03-29T11:18:58.530 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:58.530 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:58.530 INFO:teuthology.orchestra.run.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:58.530 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-03-29T11:18:58.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.543+0000 7ff291b1c640 10 monclient: shutdown 2024-03-29T11:18:58.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.543+0000 7ff291b1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:58.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.543+0000 7ff291b1c640 1 -- 172.21.15.17:0/3626919185 >> v1:172.21.15.17:6789/0 conn(0x7ff28c0b1f60 legacy=0x7ff28c09c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:58.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.543+0000 7ff291b1c640 1 -- 172.21.15.17:0/3626919185 shutdown_connections 2024-03-29T11:18:58.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.543+0000 7ff291b1c640 1 -- 172.21.15.17:0/3626919185 >> 172.21.15.17:0/3626919185 conn(0x7ff28c066b00 msgr2=0x7ff28c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:58.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.543+0000 7ff291b1c640 1 -- 172.21.15.17:0/3626919185 shutdown_connections 2024-03-29T11:18:58.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.543+0000 7ff291b1c640 1 -- 172.21.15.17:0/3626919185 wait complete. 2024-03-29T11:18:58.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.543+0000 7ff291b1c640 10 monclient: build_initial_monmap 2024-03-29T11:18:58.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.543+0000 7ff291b1c640 10 monclient: monmap: 2024-03-29T11:18:58.531 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-03-29T11:18:58.531 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:58.531 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-03-29T11:18:58.531 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-03-29T11:18:58.531 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:58.532 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-03-29T11:18:58.532 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:18:58.532 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:18:58.532 INFO:teuthology.orchestra.run.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:18:58.532 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-03-29T11:18:58.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.544+0000 7ff291b1c640 1 librados: starting msgr at 2024-03-29T11:18:58.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.544+0000 7ff291b1c640 1 librados: starting objecter 2024-03-29T11:18:58.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.544+0000 7ff291b1c640 1 Processor -- start 2024-03-29T11:18:58.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.544+0000 7ff291b1c640 1 -- start start 2024-03-29T11:18:58.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.544+0000 7ff291b1c640 1 librados: setting wanted keys 2024-03-29T11:18:58.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.544+0000 7ff291b1c640 1 librados: calling monclient init 2024-03-29T11:18:58.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.544+0000 7ff291b1c640 10 monclient: init 2024-03-29T11:18:58.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.544+0000 7ff291b1c640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:58.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.544+0000 7ff291b1c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:18:58.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.544+0000 7ff291b1c640 10 monclient(hunting): picked mon.noname-b con 0x7ff28c09c8c0 addr v1:172.21.15.94:6789/0 2024-03-29T11:18:58.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff291b1c640 10 monclient(hunting): picked mon.noname-a con 0x7ff28c0b8240 addr v1:172.21.15.17:6789/0 2024-03-29T11:18:58.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff291b1c640 10 monclient(hunting): picked mon.noname-c con 0x7ff28c132a60 addr v1:172.21.15.17:6790/0 2024-03-29T11:18:58.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff291b1c640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff28c133cd0 con 0x7ff28c0b8240 2024-03-29T11:18:58.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff291b1c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff28c1318e0 con 0x7ff28c09c8c0 2024-03-29T11:18:58.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff291b1c640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff28c1d74f0 con 0x7ff28c132a60 2024-03-29T11:18:58.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff291b1c640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:58.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff291b1c640 10 monclient(hunting): authenticate will time out at 844.749950s 2024-03-29T11:18:58.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff28b7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7ff28c09c8c0 0x7ff28c1d7a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:34592/0 (socket says 172.21.15.17:34592) 2024-03-29T11:18:58.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff28b7fe640 1 -- 172.21.15.17:0/2293587857 learned_addr learned my addr 172.21.15.17:0/2293587857 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:18:58.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff282ffd640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7ff28c0b8240 0x7ff28c132350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:58860/0 (socket says 172.21.15.17:58860) 2024-03-29T11:18:58.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 10 client.?.objecter ms_handle_connect 0x7ff28c0b8240 2024-03-29T11:18:58.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:58.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 10 client.?.objecter ms_handle_connect 0x7ff28c09c8c0 2024-03-29T11:18:58.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:58.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 10 client.?.objecter ms_handle_connect 0x7ff28c132a60 2024-03-29T11:18:58.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:58.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 1 -- 172.21.15.17:0/2293587857 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 955993251 0 0) 0x7ff28c1318e0 con 0x7ff28c09c8c0 2024-03-29T11:18:58.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:18:58.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:58.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:58.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 10 monclient(hunting): my global_id is 4192 2024-03-29T11:18:58.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 1 -- 172.21.15.17:0/2293587857 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff26c004eb0 con 0x7ff28c09c8c0 2024-03-29T11:18:58.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 1 -- 172.21.15.17:0/2293587857 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2856929339 0 0) 0x7ff28c133cd0 con 0x7ff28c0b8240 2024-03-29T11:18:58.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:18:58.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:58.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:58.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 10 monclient(hunting): my global_id is 4188 2024-03-29T11:18:58.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 1 -- 172.21.15.17:0/2293587857 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff28c1318e0 con 0x7ff28c0b8240 2024-03-29T11:18:58.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 1 -- 172.21.15.17:0/2293587857 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2048602597 0 0) 0x7ff28c1d74f0 con 0x7ff28c132a60 2024-03-29T11:18:58.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:18:58.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:58.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:58.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 10 monclient(hunting): my global_id is 4190 2024-03-29T11:18:58.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 1 -- 172.21.15.17:0/2293587857 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff28c133cd0 con 0x7ff28c132a60 2024-03-29T11:18:58.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 1 -- 172.21.15.17:0/2293587857 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1953197593 0 0) 0x7ff26c004eb0 con 0x7ff28c09c8c0 2024-03-29T11:18:58.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 1 -- 172.21.15.17:0/2293587857 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff28c1d74f0 con 0x7ff28c09c8c0 2024-03-29T11:18:58.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 1 -- 172.21.15.17:0/2293587857 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7ff27c001bd0 con 0x7ff28c09c8c0 2024-03-29T11:18:58.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:58.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:18:58.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 10 monclient(hunting): dump: 2024-03-29T11:18:58.535 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-03-29T11:18:58.535 INFO:teuthology.orchestra.run.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:58.535 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:58.535 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:58.536 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:58.536 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-03-29T11:18:58.536 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:58.536 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:58.536 INFO:teuthology.orchestra.run.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:58.536 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-03-29T11:18:58.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:58.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 1 -- 172.21.15.17:0/2293587857 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2338139654 0 0) 0x7ff28c1318e0 con 0x7ff28c0b8240 2024-03-29T11:18:58.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 1 -- 172.21.15.17:0/2293587857 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff26c004eb0 con 0x7ff28c0b8240 2024-03-29T11:18:58.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 1 -- 172.21.15.17:0/2293587857 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7ff274001340 con 0x7ff28c0b8240 2024-03-29T11:18:58.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:58.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:58.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 10 monclient(hunting): dump: 2024-03-29T11:18:58.536 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-03-29T11:18:58.537 INFO:teuthology.orchestra.run.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:58.537 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:58.537 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:58.537 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:58.537 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-03-29T11:18:58.537 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:58.537 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:58.537 INFO:teuthology.orchestra.run.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:58.537 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-03-29T11:18:58.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 1 -- 172.21.15.17:0/2293587857 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4216701847 0 0) 0x7ff28c133cd0 con 0x7ff28c132a60 2024-03-29T11:18:58.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 1 -- 172.21.15.17:0/2293587857 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff28c1318e0 con 0x7ff28c132a60 2024-03-29T11:18:58.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 1 -- 172.21.15.17:0/2293587857 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7ff284001b90 con 0x7ff28c132a60 2024-03-29T11:18:58.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:58.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:58.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 10 monclient(hunting): dump: 2024-03-29T11:18:58.538 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-03-29T11:18:58.538 INFO:teuthology.orchestra.run.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:58.538 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:58.538 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:58.538 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:58.538 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-03-29T11:18:58.538 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:58.538 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:58.538 INFO:teuthology.orchestra.run.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:58.538 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-03-29T11:18:58.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 1 -- 172.21.15.17:0/2293587857 <== mon.1 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3795679440 0 0) 0x7ff28c1d74f0 con 0x7ff28c09c8c0 2024-03-29T11:18:58.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 1 -- 172.21.15.17:0/2293587857 >> v1:172.21.15.17:6790/0 conn(0x7ff28c132a60 legacy=0x7ff28c1e1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:58.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 1 -- 172.21.15.17:0/2293587857 >> v1:172.21.15.17:6789/0 conn(0x7ff28c0b8240 legacy=0x7ff28c132350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:58.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:18:58.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 1 monclient: found mon.b 2024-03-29T11:18:58.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:58.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:18:58.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 1 -- 172.21.15.17:0/2293587857 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff28c1d8120 con 0x7ff28c09c8c0 2024-03-29T11:18:58.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 10 monclient: _finish_auth 0 2024-03-29T11:18:58.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:18:58.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff288ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:58.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff291b1c640 5 monclient: authenticate success, global_id 4192 2024-03-29T11:18:58.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff291b1c640 10 monclient: _renew_subs 2024-03-29T11:18:58.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff291b1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:18:58.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff291b1c640 1 -- 172.21.15.17:0/2293587857 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff28c1d4e60 con 0x7ff28c09c8c0 2024-03-29T11:18:58.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff291b1c640 10 client.4192.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:58.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff291b1c640 10 monclient: _renew_subs 2024-03-29T11:18:58.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff291b1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:18:58.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff291b1c640 1 -- 172.21.15.17:0/2293587857 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff28c1e3480 con 0x7ff28c09c8c0 2024-03-29T11:18:58.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.545+0000 7ff291b1c640 1 librados: init done 2024-03-29T11:18:58.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.546+0000 7ff288ff9640 1 -- 172.21.15.17:0/2293587857 <== mon.1 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff27c0047e0 con 0x7ff28c09c8c0 2024-03-29T11:18:58.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.546+0000 7ff288ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:58.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.546+0000 7ff288ff9640 1 -- 172.21.15.17:0/2293587857 <== mon.1 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7ff27c004af0 con 0x7ff28c09c8c0 2024-03-29T11:18:58.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.546+0000 7ff288ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:58.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.546+0000 7ff288ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:18:58.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.546+0000 7ff28a7fc640 4 set_mon_vals no callback set 2024-03-29T11:18:58.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.546+0000 7ff288ff9640 10 monclient: dump: 2024-03-29T11:18:58.540 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-03-29T11:18:58.540 INFO:teuthology.orchestra.run.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:58.540 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:58.540 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:58.540 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:58.540 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-03-29T11:18:58.541 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:58.541 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:58.541 INFO:teuthology.orchestra.run.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:58.541 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-03-29T11:18:58.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.546+0000 7ff288ff9640 1 -- 172.21.15.17:0/2293587857 <== mon.1 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ff27c005060 con 0x7ff28c09c8c0 2024-03-29T11:18:58.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.546+0000 7ff288ff9640 1 -- 172.21.15.17:0/2293587857 <== mon.1 v1:172.21.15.94:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7ff27c006960 con 0x7ff28c09c8c0 2024-03-29T11:18:58.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.546+0000 7ff288ff9640 10 client.4192.objecter ms_dispatch 0x7ff28c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:58.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.546+0000 7ff288ff9640 3 client.4192.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:58.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.546+0000 7ff288ff9640 3 client.4192.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:58.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.546+0000 7ff288ff9640 20 client.4192.objecter dump_active .. 0 homeless 2024-03-29T11:18:58.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.546+0000 7ff291b1c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:58.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.546+0000 7ff291b1c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:58.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.546+0000 7ff291b1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:18:58.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.546+0000 7ff291b1c640 1 -- 172.21.15.17:0/2293587857 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff270005080 con 0x7ff28c09c8c0 2024-03-29T11:18:58.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.549+0000 7ff288ff9640 1 -- 172.21.15.17:0/2293587857 <== mon.1 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ff27c006bc0 con 0x7ff28c09c8c0 2024-03-29T11:18:58.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.549+0000 7ff288ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:58.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.549+0000 7ff288ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:58.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.611+0000 7ff291b1c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:58.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.611+0000 7ff291b1c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:58.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.611+0000 7ff291b1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:18:58.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.611+0000 7ff291b1c640 1 -- 172.21.15.17:0/2293587857 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff270005330 con 0x7ff28c09c8c0 2024-03-29T11:18:58.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.612+0000 7ff288ff9640 1 -- 172.21.15.17:0/2293587857 <== mon.1 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7ff27c0063a0 con 0x7ff28c09c8c0 2024-03-29T11:18:58.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.612+0000 7ff288ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:58.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.612+0000 7ff288ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:58.590 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-29T11:18:58.590 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:58.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.613+0000 7ff291b1c640 10 librados: watch_flush enter 2024-03-29T11:18:58.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.613+0000 7ff291b1c640 10 librados: watch_flush exit 2024-03-29T11:18:58.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.613+0000 7ff291b1c640 20 client.4192.objecter shutdown clearing up homeless session... 2024-03-29T11:18:58.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.613+0000 7ff291b1c640 10 client.4192.objecter successfully canceled tick 2024-03-29T11:18:58.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.613+0000 7ff291b1c640 10 monclient: shutdown 2024-03-29T11:18:58.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.613+0000 7ff291b1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:58.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.613+0000 7ff291b1c640 1 -- 172.21.15.17:0/2293587857 >> v1:172.21.15.94:6789/0 conn(0x7ff28c09c8c0 legacy=0x7ff28c1d7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:58.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.613+0000 7ff291b1c640 1 -- 172.21.15.17:0/2293587857 shutdown_connections 2024-03-29T11:18:58.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.613+0000 7ff291b1c640 1 -- 172.21.15.17:0/2293587857 >> 172.21.15.17:0/2293587857 conn(0x7ff28c066b00 msgr2=0x7ff28c066ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:58.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.613+0000 7ff291b1c640 1 -- 172.21.15.17:0/2293587857 shutdown_connections 2024-03-29T11:18:58.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.613+0000 7ff291b1c640 1 -- 172.21.15.17:0/2293587857 wait complete. 2024-03-29T11:18:58.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-03-29T11:18:58.613+0000 7ff291b1c640 1 librados: shutdown 2024-03-29T11:18:58.599 INFO:tasks.ceph.mgr.y.smithi017.stderr:2024-03-29T11:18:58.622+0000 7f6f3d22e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:18:58.600 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T11:18:58.600 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T11:18:58.632 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-03-29T11:18:58.656+0000 7fcdfee3c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:18:58.646 INFO:tasks.ceph.mgr.y.smithi017.stderr:2024-03-29T11:18:58.669+0000 7f6f3d22e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:18:58.678 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-03-29T11:18:58.701+0000 7fcdfee3c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:18:58.762 INFO:tasks.ceph.mgr.y.smithi017.stderr:2024-03-29T11:18:58.786+0000 7f6f3d22e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:18:58.791 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-03-29T11:18:58.814+0000 7fcdfee3c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:18:58.861 INFO:tasks.ceph.mgr.y.smithi017.stderr:2024-03-29T11:18:58.884+0000 7f6f3d22e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:18:58.884 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-03-29T11:18:58.907+0000 7fcdfee3c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:18:58.986 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-03-29T11:18:59.010+0000 7f4a99e0b640 -1 Falling back to public interface 2024-03-29T11:18:58.991 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-03-29T11:18:59.014+0000 7f558600d640 -1 Falling back to public interface 2024-03-29T11:18:58.995 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-03-29T11:18:59.018+0000 7f39ec412640 -1 Falling back to public interface 2024-03-29T11:18:58.999 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-03-29T11:18:59.022+0000 7fec84615640 -1 Falling back to public interface 2024-03-29T11:18:59.006 INFO:tasks.ceph.osd.3.smithi017.stderr:2024-03-29T11:18:59.029+0000 7f8f95cd9640 -1 Falling back to public interface 2024-03-29T11:18:59.007 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-03-29T11:18:59.030+0000 7f3812099640 -1 Falling back to public interface 2024-03-29T11:18:59.011 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-03-29T11:18:59.035+0000 7f3909338640 -1 Falling back to public interface 2024-03-29T11:18:59.015 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-03-29T11:18:59.038+0000 7f8923a0c640 -1 Falling back to public interface 2024-03-29T11:18:59.024 INFO:tasks.ceph.mgr.y.smithi017.stderr:2024-03-29T11:18:59.048+0000 7f6f3d22e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:18:59.043 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-03-29T11:18:59.066+0000 7fcdfee3c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:18:59.068 INFO:tasks.ceph.mgr.y.smithi017.stderr:2024-03-29T11:18:59.092+0000 7f6f3d22e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:18:59.085 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-03-29T11:18:59.108+0000 7fcdfee3c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:18:59.229 INFO:tasks.ceph.mgr.y.smithi017.stderr:2024-03-29T11:18:59.252+0000 7f6f3d22e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:18:59.239 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-03-29T11:18:59.262+0000 7fcdfee3c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:18:59.406 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:59.496 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.519+0000 7fd25ea33640 10 monclient: get_monmap_and_config 2024-03-29T11:18:59.496 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.519+0000 7fd25ea33640 10 monclient: build_initial_monmap 2024-03-29T11:18:59.496 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.519+0000 7fd25ea33640 10 monclient: monmap: 2024-03-29T11:18:59.496 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:18:59.496 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:59.497 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:59.520514+0000 2024-03-29T11:18:59.497 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:59.520514+0000 2024-03-29T11:18:59.497 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:59.497 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:18:59.497 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:18:59.497 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:18:59.497 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:18:59.497 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:18:59.505 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.520+0000 7fd25ea33640 1 Processor -- start 2024-03-29T11:18:59.506 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.520+0000 7fd25ea33640 1 -- start start 2024-03-29T11:18:59.506 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.520+0000 7fd25ea33640 10 monclient: init 2024-03-29T11:18:59.506 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.520+0000 7fd25ea33640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:59.506 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.520+0000 7fd25ea33640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:18:59.506 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.520+0000 7fd25ea33640 10 monclient(hunting): picked mon.noname-b con 0x7fd2581457f0 addr v1:172.21.15.17:6790/0 2024-03-29T11:18:59.506 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.520+0000 7fd25ea33640 10 monclient(hunting): picked mon.noname-c con 0x7fd25813f650 addr v1:172.21.15.94:6789/0 2024-03-29T11:18:59.506 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.520+0000 7fd25ea33640 10 monclient(hunting): picked mon.noname-a con 0x7fd2580a9d70 addr v1:172.21.15.17:6789/0 2024-03-29T11:18:59.506 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.520+0000 7fd25ea33640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2580aaef0 con 0x7fd2580a9d70 2024-03-29T11:18:59.506 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.520+0000 7fd25ea33640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd258145bd0 con 0x7fd25813f650 2024-03-29T11:18:59.506 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.520+0000 7fd25ea33640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd258145eb0 con 0x7fd2581457f0 2024-03-29T11:18:59.506 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.520+0000 7fd25ea33640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:59.507 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.520+0000 7fd25ea33640 10 monclient(hunting): authenticate will time out at 845.725752s 2024-03-29T11:18:59.507 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.521+0000 7fd257fff640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7fd2581457f0 0x7fd25813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:42360/0 (socket says 172.21.15.17:42360) 2024-03-29T11:18:59.507 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.521+0000 7fd25cfd0640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fd2580a9d70 0x7fd25814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:58866/0 (socket says 172.21.15.17:58866) 2024-03-29T11:18:59.507 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.521+0000 7fd257fff640 1 -- 172.21.15.17:0/2963892512 learned_addr learned my addr 172.21.15.17:0/2963892512 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:18:59.507 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.521+0000 7fd256ffd640 1 -- 172.21.15.17:0/2963892512 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4239101598 0 0) 0x7fd258145eb0 con 0x7fd2581457f0 2024-03-29T11:18:59.507 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.521+0000 7fd256ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:18:59.507 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.521+0000 7fd256ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:59.507 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.521+0000 7fd256ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:59.507 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.521+0000 7fd256ffd640 10 monclient(hunting): my global_id is 4193 2024-03-29T11:18:59.507 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.521+0000 7fd256ffd640 1 -- 172.21.15.17:0/2963892512 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd240003520 con 0x7fd2581457f0 2024-03-29T11:18:59.507 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.521+0000 7fd256ffd640 1 -- 172.21.15.17:0/2963892512 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2246075880 0 0) 0x7fd2580aaef0 con 0x7fd2580a9d70 2024-03-29T11:18:59.507 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.521+0000 7fd256ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:18:59.507 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.521+0000 7fd256ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:59.507 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.521+0000 7fd256ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:59.507 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.521+0000 7fd256ffd640 10 monclient(hunting): my global_id is 4191 2024-03-29T11:18:59.507 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.521+0000 7fd256ffd640 1 -- 172.21.15.17:0/2963892512 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd258145eb0 con 0x7fd2580a9d70 2024-03-29T11:18:59.508 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.521+0000 7fd256ffd640 1 -- 172.21.15.17:0/2963892512 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 781540016 0 0) 0x7fd258145bd0 con 0x7fd25813f650 2024-03-29T11:18:59.508 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.521+0000 7fd256ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:18:59.508 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.521+0000 7fd256ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:59.508 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.521+0000 7fd256ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:59.508 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.521+0000 7fd256ffd640 10 monclient(hunting): my global_id is 4195 2024-03-29T11:18:59.508 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.521+0000 7fd256ffd640 1 -- 172.21.15.17:0/2963892512 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2580aaef0 con 0x7fd25813f650 2024-03-29T11:18:59.508 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.521+0000 7fd256ffd640 1 -- 172.21.15.17:0/2963892512 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1897928499 0 0) 0x7fd240003520 con 0x7fd2581457f0 2024-03-29T11:18:59.508 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.521+0000 7fd256ffd640 1 -- 172.21.15.17:0/2963892512 >> v1:172.21.15.94:6789/0 conn(0x7fd25813f650 legacy=0x7fd2580b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:59.508 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.521+0000 7fd256ffd640 1 -- 172.21.15.17:0/2963892512 >> v1:172.21.15.17:6789/0 conn(0x7fd2580a9d70 legacy=0x7fd25814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:59.508 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.521+0000 7fd256ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:18:59.508 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.521+0000 7fd256ffd640 1 monclient: found mon.noname-b 2024-03-29T11:18:59.508 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.521+0000 7fd256ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:59.508 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.521+0000 7fd256ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.17:6790/0 2024-03-29T11:18:59.508 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.521+0000 7fd256ffd640 1 -- 172.21.15.17:0/2963892512 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd258146340 con 0x7fd2581457f0 2024-03-29T11:18:59.509 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.521+0000 7fd256ffd640 10 monclient: _finish_auth 0 2024-03-29T11:18:59.509 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.521+0000 7fd256ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:18:59.509 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.521+0000 7fd256ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:59.509 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.521+0000 7fd256ffd640 1 -- 172.21.15.17:0/2963892512 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fd2480015c0 con 0x7fd2581457f0 2024-03-29T11:18:59.509 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.521+0000 7fd256ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:59.509 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.521+0000 7fd256ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:18:59.509 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.521+0000 7fd256ffd640 10 monclient: dump: 2024-03-29T11:18:59.509 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:18:59.509 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:59.509 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:59.509 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:59.509 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:59.509 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:18:59.509 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:59.509 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:59.510 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:59.510 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:18:59.510 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.521+0000 7fd25ea33640 5 monclient: authenticate success, global_id 4193 2024-03-29T11:18:59.510 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.521+0000 7fd25ea33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:59.510 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.521+0000 7fd256ffd640 1 -- 172.21.15.17:0/2963892512 <== mon.2 v1:172.21.15.17:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd248001720 con 0x7fd2581457f0 2024-03-29T11:18:59.510 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.521+0000 7fd256ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:59.510 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.521+0000 7fd256ffd640 1 -- 172.21.15.17:0/2963892512 <== mon.2 v1:172.21.15.17:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fd248001b70 con 0x7fd2581457f0 2024-03-29T11:18:59.510 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.521+0000 7fd25ea33640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:59.510 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.521+0000 7fd25ea33640 4 set_mon_vals no callback set 2024-03-29T11:18:59.510 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.521+0000 7fd256ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:59.510 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.521+0000 7fd256ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:59.510 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.521+0000 7fd256ffd640 10 monclient: dump: 2024-03-29T11:18:59.510 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:18:59.510 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:59.511 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:59.511 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:59.511 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:59.511 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:18:59.511 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:59.511 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:59.511 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:59.511 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:18:59.511 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.522+0000 7fd25ea33640 10 monclient: shutdown 2024-03-29T11:18:59.511 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.522+0000 7fd25ea33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:59.511 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.522+0000 7fd25ea33640 1 -- 172.21.15.17:0/2963892512 >> v1:172.21.15.17:6790/0 conn(0x7fd2581457f0 legacy=0x7fd25813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:59.511 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.522+0000 7fd25ea33640 1 -- 172.21.15.17:0/2963892512 shutdown_connections 2024-03-29T11:18:59.511 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.522+0000 7fd25ea33640 1 -- 172.21.15.17:0/2963892512 >> 172.21.15.17:0/2963892512 conn(0x7fd258066a00 msgr2=0x7fd258066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:59.511 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.522+0000 7fd25ea33640 1 -- 172.21.15.17:0/2963892512 shutdown_connections 2024-03-29T11:18:59.512 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.522+0000 7fd25ea33640 1 -- 172.21.15.17:0/2963892512 wait complete. 2024-03-29T11:18:59.512 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.522+0000 7fd25ea33640 10 monclient: build_initial_monmap 2024-03-29T11:18:59.512 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.522+0000 7fd25ea33640 10 monclient: monmap: 2024-03-29T11:18:59.512 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:18:59.512 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:59.512 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:18:59.512 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:18:59.512 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:59.512 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:18:59.512 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:18:59.512 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:18:59.512 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:18:59.512 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:18:59.512 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.523+0000 7fd25ea33640 1 librados: starting msgr at 2024-03-29T11:18:59.513 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.523+0000 7fd25ea33640 1 librados: starting objecter 2024-03-29T11:18:59.513 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.523+0000 7fd25ea33640 1 Processor -- start 2024-03-29T11:18:59.513 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.523+0000 7fd25ea33640 1 -- start start 2024-03-29T11:18:59.513 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.523+0000 7fd25ea33640 1 librados: setting wanted keys 2024-03-29T11:18:59.513 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.523+0000 7fd25ea33640 1 librados: calling monclient init 2024-03-29T11:18:59.513 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.523+0000 7fd25ea33640 10 monclient: init 2024-03-29T11:18:59.513 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.523+0000 7fd25ea33640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:59.513 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.523+0000 7fd25ea33640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:18:59.513 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.523+0000 7fd25ea33640 10 monclient(hunting): picked mon.noname-c con 0x7fd2580a9d70 addr v1:172.21.15.17:6790/0 2024-03-29T11:18:59.513 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.523+0000 7fd25ea33640 10 monclient(hunting): picked mon.noname-a con 0x7fd25813f650 addr v1:172.21.15.17:6789/0 2024-03-29T11:18:59.513 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.523+0000 7fd25ea33640 10 monclient(hunting): picked mon.noname-b con 0x7fd2581457f0 addr v1:172.21.15.94:6789/0 2024-03-29T11:18:59.513 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.523+0000 7fd25ea33640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2580ad360 con 0x7fd25813f650 2024-03-29T11:18:59.513 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.523+0000 7fd25ea33640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2580addb0 con 0x7fd2581457f0 2024-03-29T11:18:59.513 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd25ea33640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2580ae080 con 0x7fd2580a9d70 2024-03-29T11:18:59.514 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd25ea33640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:59.514 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd25ea33640 10 monclient(hunting): authenticate will time out at 845.728890s 2024-03-29T11:18:59.514 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd2577fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fd25813f650 0x7fd2581da960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:58880/0 (socket says 172.21.15.17:58880) 2024-03-29T11:18:59.514 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd2577fe640 1 -- 172.21.15.17:0/888934804 learned_addr learned my addr 172.21.15.17:0/888934804 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:18:59.514 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 10 client.?.objecter ms_handle_connect 0x7fd2580a9d70 2024-03-29T11:18:59.514 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:59.514 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 10 client.?.objecter ms_handle_connect 0x7fd25813f650 2024-03-29T11:18:59.514 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:59.514 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 10 client.?.objecter ms_handle_connect 0x7fd2581457f0 2024-03-29T11:18:59.514 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:59.514 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 1 -- 172.21.15.17:0/888934804 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2240971016 0 0) 0x7fd2580ae080 con 0x7fd2580a9d70 2024-03-29T11:18:59.514 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:18:59.514 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:59.514 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:59.515 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 10 monclient(hunting): my global_id is 4196 2024-03-29T11:18:59.515 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 1 -- 172.21.15.17:0/888934804 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd244004eb0 con 0x7fd2580a9d70 2024-03-29T11:18:59.515 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 1 -- 172.21.15.17:0/888934804 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3408773845 0 0) 0x7fd2580ad360 con 0x7fd25813f650 2024-03-29T11:18:59.515 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:18:59.515 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:59.515 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:59.515 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 10 monclient(hunting): my global_id is 4194 2024-03-29T11:18:59.515 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 1 -- 172.21.15.17:0/888934804 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2580ae080 con 0x7fd25813f650 2024-03-29T11:18:59.515 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 1 -- 172.21.15.17:0/888934804 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1672709791 0 0) 0x7fd2580addb0 con 0x7fd2581457f0 2024-03-29T11:18:59.515 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:18:59.515 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:59.515 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:59.515 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 10 monclient(hunting): my global_id is 4198 2024-03-29T11:18:59.515 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 1 -- 172.21.15.17:0/888934804 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2580ad360 con 0x7fd2581457f0 2024-03-29T11:18:59.515 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 1 -- 172.21.15.17:0/888934804 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2203761606 0 0) 0x7fd244004eb0 con 0x7fd2580a9d70 2024-03-29T11:18:59.516 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 1 -- 172.21.15.17:0/888934804 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2580addb0 con 0x7fd2580a9d70 2024-03-29T11:18:59.516 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 1 -- 172.21.15.17:0/888934804 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 779773780 0 0) 0x7fd2580ae080 con 0x7fd25813f650 2024-03-29T11:18:59.516 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 1 -- 172.21.15.17:0/888934804 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd244004eb0 con 0x7fd25813f650 2024-03-29T11:18:59.516 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 1 -- 172.21.15.17:0/888934804 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fd248001af0 con 0x7fd2580a9d70 2024-03-29T11:18:59.516 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:59.516 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:18:59.516 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 10 monclient(hunting): dump: 2024-03-29T11:18:59.516 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:18:59.516 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:59.516 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:59.516 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:59.516 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:59.516 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:18:59.516 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:59.517 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:59.517 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:59.517 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:18:59.517 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:59.517 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 1 -- 172.21.15.17:0/888934804 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fd24c001970 con 0x7fd25813f650 2024-03-29T11:18:59.517 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:59.517 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:59.517 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 10 monclient(hunting): dump: 2024-03-29T11:18:59.517 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:18:59.517 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:59.517 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:59.517 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:59.517 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:59.517 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:18:59.517 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:59.518 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:59.518 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:59.518 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:18:59.518 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 1 -- 172.21.15.17:0/888934804 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2099451574 0 0) 0x7fd2580ad360 con 0x7fd2581457f0 2024-03-29T11:18:59.518 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 1 -- 172.21.15.17:0/888934804 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2580ae080 con 0x7fd2581457f0 2024-03-29T11:18:59.518 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 1 -- 172.21.15.17:0/888934804 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fd250001c30 con 0x7fd2581457f0 2024-03-29T11:18:59.518 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:59.518 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:18:59.518 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 10 monclient(hunting): dump: 2024-03-29T11:18:59.518 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:18:59.518 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:59.518 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:59.518 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:59.518 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:59.519 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:18:59.519 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:59.519 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:59.519 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:59.519 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:18:59.519 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 1 -- 172.21.15.17:0/888934804 <== mon.0 v1:172.21.15.17:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 64031193 0 0) 0x7fd244004eb0 con 0x7fd25813f650 2024-03-29T11:18:59.519 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 1 -- 172.21.15.17:0/888934804 >> v1:172.21.15.17:6790/0 conn(0x7fd2580a9d70 legacy=0x7fd2580b09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:59.519 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 1 -- 172.21.15.17:0/888934804 >> v1:172.21.15.94:6789/0 conn(0x7fd2581457f0 legacy=0x7fd2581db070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:59.519 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:18:59.519 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 1 monclient: found mon.a 2024-03-29T11:18:59.519 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:59.519 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:18:59.519 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 1 -- 172.21.15.17:0/888934804 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2580acbb0 con 0x7fd25813f650 2024-03-29T11:18:59.519 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 10 monclient: _finish_auth 0 2024-03-29T11:18:59.520 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:18:59.520 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd254ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:59.520 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd25ea33640 5 monclient: authenticate success, global_id 4194 2024-03-29T11:18:59.520 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd25ea33640 10 monclient: _renew_subs 2024-03-29T11:18:59.520 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd25ea33640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:18:59.520 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd25ea33640 1 -- 172.21.15.17:0/888934804 --> v1:172.21.15.17:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2580aced0 con 0x7fd25813f650 2024-03-29T11:18:59.520 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd25ea33640 10 client.4194.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:59.520 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd25ea33640 10 monclient: _renew_subs 2024-03-29T11:18:59.520 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd25ea33640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:18:59.520 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.524+0000 7fd25ea33640 1 -- 172.21.15.17:0/888934804 --> v1:172.21.15.17:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2581424c0 con 0x7fd25813f650 2024-03-29T11:18:59.520 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.525+0000 7fd25ea33640 1 librados: init done 2024-03-29T11:18:59.520 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.525+0000 7fd254ff9640 1 -- 172.21.15.17:0/888934804 <== mon.0 v1:172.21.15.17:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd24c0035f0 con 0x7fd25813f650 2024-03-29T11:18:59.520 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.525+0000 7fd254ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:59.521 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.525+0000 7fd254ff9640 1 -- 172.21.15.17:0/888934804 <== mon.0 v1:172.21.15.17:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fd24c0039c0 con 0x7fd25813f650 2024-03-29T11:18:59.521 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.525+0000 7fd254ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:59.521 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.525+0000 7fd254ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:59.521 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.525+0000 7fd2567fc640 4 set_mon_vals no callback set 2024-03-29T11:18:59.521 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.525+0000 7fd254ff9640 10 monclient: dump: 2024-03-29T11:18:59.521 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:18:59.521 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:18:59.521 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:59.521 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:18:59.521 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:59.521 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:18:59.521 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:18:59.521 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:18:59.521 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:18:59.521 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:18:59.522 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.525+0000 7fd254ff9640 1 -- 172.21.15.17:0/888934804 <== mon.0 v1:172.21.15.17:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fd24c003f70 con 0x7fd25813f650 2024-03-29T11:18:59.522 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.525+0000 7fd254ff9640 1 -- 172.21.15.17:0/888934804 <== mon.0 v1:172.21.15.17:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7fd24c0068d0 con 0x7fd25813f650 2024-03-29T11:18:59.522 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.525+0000 7fd254ff9640 10 client.4194.objecter ms_dispatch 0x7fd258000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:59.522 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.525+0000 7fd254ff9640 3 client.4194.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:59.522 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.525+0000 7fd254ff9640 3 client.4194.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:59.522 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.525+0000 7fd254ff9640 20 client.4194.objecter dump_active .. 0 homeless 2024-03-29T11:18:59.522 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.526+0000 7fd25ea33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:59.522 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.526+0000 7fd25ea33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:59.522 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.526+0000 7fd25ea33640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:18:59.522 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.526+0000 7fd25ea33640 1 -- 172.21.15.17:0/888934804 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd238005080 con 0x7fd25813f650 2024-03-29T11:18:59.522 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.528+0000 7fd254ff9640 1 -- 172.21.15.17:0/888934804 <== mon.0 v1:172.21.15.17:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fd24c0043b0 con 0x7fd25813f650 2024-03-29T11:18:59.522 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.528+0000 7fd254ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:59.522 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.528+0000 7fd254ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:59.567 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.590+0000 7fd25ea33640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:59.567 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.590+0000 7fd25ea33640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:59.568 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.590+0000 7fd25ea33640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:18:59.568 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.590+0000 7fd25ea33640 1 -- 172.21.15.17:0/888934804 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd238005330 con 0x7fd25813f650 2024-03-29T11:18:59.568 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.591+0000 7fd254ff9640 1 -- 172.21.15.17:0/888934804 <== mon.0 v1:172.21.15.17:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7fd24c006b10 con 0x7fd25813f650 2024-03-29T11:18:59.568 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.591+0000 7fd254ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:59.568 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.591+0000 7fd254ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:59.569 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:18:59.569 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:59.570 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.592+0000 7fd25ea33640 10 librados: watch_flush enter 2024-03-29T11:18:59.570 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.592+0000 7fd25ea33640 10 librados: watch_flush exit 2024-03-29T11:18:59.570 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.592+0000 7fd25ea33640 20 client.4194.objecter shutdown clearing up homeless session... 2024-03-29T11:18:59.570 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.592+0000 7fd25ea33640 10 client.4194.objecter successfully canceled tick 2024-03-29T11:18:59.570 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.592+0000 7fd25ea33640 10 monclient: shutdown 2024-03-29T11:18:59.570 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.592+0000 7fd25ea33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:59.570 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.592+0000 7fd25ea33640 1 -- 172.21.15.17:0/888934804 >> v1:172.21.15.17:6789/0 conn(0x7fd25813f650 legacy=0x7fd2581da960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:59.570 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.592+0000 7fd25ea33640 1 -- 172.21.15.17:0/888934804 shutdown_connections 2024-03-29T11:18:59.570 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.592+0000 7fd25ea33640 1 -- 172.21.15.17:0/888934804 >> 172.21.15.17:0/888934804 conn(0x7fd258066a00 msgr2=0x7fd25813aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:59.570 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.592+0000 7fd25ea33640 1 -- 172.21.15.17:0/888934804 shutdown_connections 2024-03-29T11:18:59.570 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.592+0000 7fd25ea33640 1 -- 172.21.15.17:0/888934804 wait complete. 2024-03-29T11:18:59.570 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:18:59.592+0000 7fd25ea33640 1 librados: shutdown 2024-03-29T11:18:59.578 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:00.073 INFO:tasks.ceph.mgr.y.smithi017.stderr:2024-03-29T11:19:00.096+0000 7f6f3d22e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:19:00.359 INFO:tasks.ceph.mgr.y.smithi017.stderr:2024-03-29T11:19:00.382+0000 7f6f3d22e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:19:00.368 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-03-29T11:19:00.392+0000 7fcdfee3c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:19:00.412 INFO:tasks.ceph.mgr.y.smithi017.stderr:2024-03-29T11:19:00.436+0000 7f6f3d22e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:19:00.532 INFO:tasks.ceph.mgr.y.smithi017.stderr:2024-03-29T11:19:00.555+0000 7f6f3d22e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:19:00.644 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-03-29T11:19:00.667+0000 7fcdfee3c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:19:00.697 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-03-29T11:19:00.720+0000 7fcdfee3c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:19:00.810 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-03-29T11:19:00.833+0000 7fcdfee3c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:19:01.544 INFO:tasks.ceph.mgr.y.smithi017.stderr:2024-03-29T11:19:01.567+0000 7f6edd7dc640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T11:19:01.545 INFO:tasks.ceph.mgr.y.smithi017.stderr:2024-03-29T11:19:01.568+0000 7f6edd7dc640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T11:19:02.453 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-03-29T11:19:02.476+0000 7f558600d640 -1 osd.5 0 log_to_monitors true 2024-03-29T11:19:02.482 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-03-29T11:19:02.505+0000 7f4a99e0b640 -1 osd.0 0 log_to_monitors true 2024-03-29T11:19:02.509 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-03-29T11:19:02.532+0000 7f39ec412640 -1 osd.6 0 log_to_monitors true 2024-03-29T11:19:02.511 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-03-29T11:19:02.535+0000 7f3812099640 -1 osd.7 0 log_to_monitors true 2024-03-29T11:19:02.521 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-03-29T11:19:02.545+0000 7f3909338640 -1 osd.4 0 log_to_monitors true 2024-03-29T11:19:02.536 INFO:tasks.ceph.osd.3.smithi017.stderr:2024-03-29T11:19:02.559+0000 7f8f95cd9640 -1 osd.3 0 log_to_monitors true 2024-03-29T11:19:02.538 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-03-29T11:19:02.562+0000 7f8923a0c640 -1 osd.2 0 log_to_monitors true 2024-03-29T11:19:02.549 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-03-29T11:19:02.572+0000 7fec84615640 -1 osd.1 0 log_to_monitors true 2024-03-29T11:19:06.385 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:06.531 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.554+0000 7fba2a02b640 10 monclient: get_monmap_and_config 2024-03-29T11:19:06.531 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.554+0000 7fba2a02b640 10 monclient: build_initial_monmap 2024-03-29T11:19:06.531 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.554+0000 7fba2a02b640 10 monclient: monmap: 2024-03-29T11:19:06.531 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:19:06.531 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:19:06.531 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:19:06.555126+0000 2024-03-29T11:19:06.531 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:19:06.555126+0000 2024-03-29T11:19:06.531 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:06.532 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:06.532 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:19:06.532 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:19:06.532 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:19:06.532 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:06.532 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.555+0000 7fba2a02b640 1 Processor -- start 2024-03-29T11:19:06.532 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.555+0000 7fba2a02b640 1 -- start start 2024-03-29T11:19:06.532 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.555+0000 7fba2a02b640 10 monclient: init 2024-03-29T11:19:06.533 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.556+0000 7fba2a02b640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:06.533 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.556+0000 7fba2a02b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:06.534 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.556+0000 7fba2a02b640 10 monclient(hunting): picked mon.noname-a con 0x7fba240b4ba0 addr v1:172.21.15.17:6789/0 2024-03-29T11:19:06.534 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.556+0000 7fba2a02b640 10 monclient(hunting): picked mon.noname-c con 0x7fba240b2520 addr v1:172.21.15.94:6789/0 2024-03-29T11:19:06.534 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.556+0000 7fba2a02b640 10 monclient(hunting): picked mon.noname-b con 0x7fba2414fb30 addr v1:172.21.15.17:6790/0 2024-03-29T11:19:06.534 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.556+0000 7fba2a02b640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba2406fa10 con 0x7fba240b4ba0 2024-03-29T11:19:06.534 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.556+0000 7fba2a02b640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba240b1780 con 0x7fba240b2520 2024-03-29T11:19:06.534 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.556+0000 7fba2a02b640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba2406e500 con 0x7fba2414fb30 2024-03-29T11:19:06.534 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.556+0000 7fba2a02b640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:06.534 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.556+0000 7fba2a02b640 10 monclient(hunting): authenticate will time out at 852.761780s 2024-03-29T11:19:06.534 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.557+0000 7fba29029640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fba240b4ba0 0x7fba241438d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:59214/0 (socket says 172.21.15.17:59214) 2024-03-29T11:19:06.534 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.557+0000 7fba29029640 1 -- 172.21.15.17:0/1906259026 learned_addr learned my addr 172.21.15.17:0/1906259026 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:19:06.536 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.557+0000 7fba17fff640 1 -- 172.21.15.17:0/1906259026 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1504468892 0 0) 0x7fba2406fa10 con 0x7fba240b4ba0 2024-03-29T11:19:06.536 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.557+0000 7fba17fff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:06.536 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.557+0000 7fba17fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:06.536 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.557+0000 7fba17fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:06.536 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.557+0000 7fba17fff640 10 monclient(hunting): my global_id is 4236 2024-03-29T11:19:06.537 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.557+0000 7fba17fff640 1 -- 172.21.15.17:0/1906259026 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba10003520 con 0x7fba240b4ba0 2024-03-29T11:19:06.537 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.558+0000 7fba17fff640 1 -- 172.21.15.17:0/1906259026 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3202534115 0 0) 0x7fba240b1780 con 0x7fba240b2520 2024-03-29T11:19:06.537 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.558+0000 7fba17fff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:06.537 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.558+0000 7fba17fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:06.537 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.558+0000 7fba17fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:06.537 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.558+0000 7fba17fff640 10 monclient(hunting): my global_id is 4243 2024-03-29T11:19:06.537 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.558+0000 7fba17fff640 1 -- 172.21.15.17:0/1906259026 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba2406fa10 con 0x7fba240b2520 2024-03-29T11:19:06.537 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.558+0000 7fba17fff640 1 -- 172.21.15.17:0/1906259026 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2078750062 0 0) 0x7fba10003520 con 0x7fba240b4ba0 2024-03-29T11:19:06.537 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.558+0000 7fba17fff640 1 -- 172.21.15.17:0/1906259026 >> v1:172.21.15.17:6790/0 conn(0x7fba2414fb30 legacy=0x7fba24151f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:06.537 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.558+0000 7fba17fff640 1 -- 172.21.15.17:0/1906259026 >> v1:172.21.15.94:6789/0 conn(0x7fba240b2520 legacy=0x7fba2406ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:06.537 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.558+0000 7fba17fff640 10 monclient: _finish_hunting 0 2024-03-29T11:19:06.537 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.558+0000 7fba17fff640 1 monclient: found mon.noname-a 2024-03-29T11:19:06.537 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.558+0000 7fba17fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:06.537 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.558+0000 7fba17fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.17:6789/0 2024-03-29T11:19:06.538 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.558+0000 7fba17fff640 1 -- 172.21.15.17:0/1906259026 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba2406e690 con 0x7fba240b4ba0 2024-03-29T11:19:06.538 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.558+0000 7fba17fff640 10 monclient: _finish_auth 0 2024-03-29T11:19:06.538 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.558+0000 7fba17fff640 10 monclient: _check_auth_tickets 2024-03-29T11:19:06.538 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.558+0000 7fba17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:06.538 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.558+0000 7fba17fff640 1 -- 172.21.15.17:0/1906259026 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fba18001690 con 0x7fba240b4ba0 2024-03-29T11:19:06.538 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.558+0000 7fba17fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:06.538 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.558+0000 7fba17fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:06.538 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.558+0000 7fba17fff640 10 monclient: dump: 2024-03-29T11:19:06.538 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:19:06.538 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:19:06.538 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:06.538 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:06.538 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:06.538 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:06.539 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:19:06.539 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:19:06.539 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:19:06.539 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:06.539 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.558+0000 7fba2a02b640 5 monclient: authenticate success, global_id 4236 2024-03-29T11:19:06.539 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.558+0000 7fba2a02b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:06.539 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.558+0000 7fba17fff640 1 -- 172.21.15.17:0/1906259026 <== mon.0 v1:172.21.15.17:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba180018b0 con 0x7fba240b4ba0 2024-03-29T11:19:06.539 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.559+0000 7fba17fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:06.546 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.559+0000 7fba17fff640 1 -- 172.21.15.17:0/1906259026 <== mon.0 v1:172.21.15.17:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fba18001bc0 con 0x7fba240b4ba0 2024-03-29T11:19:06.546 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.559+0000 7fba17fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:06.546 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.559+0000 7fba17fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:06.547 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.559+0000 7fba17fff640 10 monclient: dump: 2024-03-29T11:19:06.547 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:19:06.547 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:19:06.547 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:06.547 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:06.547 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:06.547 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:06.547 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:19:06.547 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:19:06.547 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:19:06.547 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:06.547 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.559+0000 7fba2a02b640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:06.547 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.559+0000 7fba2a02b640 4 set_mon_vals no callback set 2024-03-29T11:19:06.547 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.560+0000 7fba2a02b640 10 monclient: shutdown 2024-03-29T11:19:06.547 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.560+0000 7fba2a02b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:06.548 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.560+0000 7fba2a02b640 1 -- 172.21.15.17:0/1906259026 >> v1:172.21.15.17:6789/0 conn(0x7fba240b4ba0 legacy=0x7fba241438d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:06.548 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.560+0000 7fba2a02b640 1 -- 172.21.15.17:0/1906259026 shutdown_connections 2024-03-29T11:19:06.548 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.560+0000 7fba2a02b640 1 -- 172.21.15.17:0/1906259026 >> 172.21.15.17:0/1906259026 conn(0x7fba24066be0 msgr2=0x7fba24066ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:06.548 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.561+0000 7fba2a02b640 1 -- 172.21.15.17:0/1906259026 shutdown_connections 2024-03-29T11:19:06.548 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.561+0000 7fba2a02b640 1 -- 172.21.15.17:0/1906259026 wait complete. 2024-03-29T11:19:06.548 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.562+0000 7fba2a02b640 10 monclient: build_initial_monmap 2024-03-29T11:19:06.548 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.562+0000 7fba2a02b640 10 monclient: monmap: 2024-03-29T11:19:06.548 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:19:06.548 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:06.548 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:19:06.548 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:19:06.548 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:06.548 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:06.548 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:19:06.548 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:19:06.548 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:19:06.549 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:06.549 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.562+0000 7fba2a02b640 1 librados: starting msgr at 2024-03-29T11:19:06.549 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.562+0000 7fba2a02b640 1 librados: starting objecter 2024-03-29T11:19:06.549 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.562+0000 7fba2a02b640 1 Processor -- start 2024-03-29T11:19:06.549 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.563+0000 7fba2a02b640 1 -- start start 2024-03-29T11:19:06.549 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.563+0000 7fba2a02b640 1 librados: setting wanted keys 2024-03-29T11:19:06.549 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.563+0000 7fba2a02b640 1 librados: calling monclient init 2024-03-29T11:19:06.549 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.563+0000 7fba2a02b640 10 monclient: init 2024-03-29T11:19:06.549 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.563+0000 7fba2a02b640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:06.549 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.563+0000 7fba2a02b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:06.549 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.563+0000 7fba2a02b640 10 monclient(hunting): picked mon.noname-c con 0x7fba240b2520 addr v1:172.21.15.17:6790/0 2024-03-29T11:19:06.549 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.563+0000 7fba2a02b640 10 monclient(hunting): picked mon.noname-b con 0x7fba240b4ba0 addr v1:172.21.15.94:6789/0 2024-03-29T11:19:06.549 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.564+0000 7fba2a02b640 10 monclient(hunting): picked mon.noname-a con 0x7fba2414fb30 addr v1:172.21.15.17:6789/0 2024-03-29T11:19:06.550 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.564+0000 7fba29029640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7fba240b2520 0x7fba2413ab40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:40906/0 (socket says 172.21.15.17:40906) 2024-03-29T11:19:06.550 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.564+0000 7fba29029640 1 -- 172.21.15.17:0/416269066 learned_addr learned my addr 172.21.15.17:0/416269066 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:19:06.550 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.564+0000 7fba2a02b640 1 -- 172.21.15.17:0/416269066 --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba24148070 con 0x7fba2414fb30 2024-03-29T11:19:06.550 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.564+0000 7fba2a02b640 1 -- 172.21.15.17:0/416269066 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba24148350 con 0x7fba240b4ba0 2024-03-29T11:19:06.550 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.564+0000 7fba2a02b640 1 -- 172.21.15.17:0/416269066 --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba24148630 con 0x7fba240b2520 2024-03-29T11:19:06.550 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.564+0000 7fba2a02b640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:06.550 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.564+0000 7fba2a02b640 10 monclient(hunting): authenticate will time out at 852.769480s 2024-03-29T11:19:06.551 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.564+0000 7fba15ffb640 10 client.?.objecter ms_handle_connect 0x7fba240b2520 2024-03-29T11:19:06.551 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.564+0000 7fba15ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:06.551 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.564+0000 7fba15ffb640 10 client.?.objecter ms_handle_connect 0x7fba240b4ba0 2024-03-29T11:19:06.551 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.564+0000 7fba15ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:06.551 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.565+0000 7fba15ffb640 10 client.?.objecter ms_handle_connect 0x7fba2414fb30 2024-03-29T11:19:06.551 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.565+0000 7fba15ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:06.551 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.565+0000 7fba15ffb640 1 -- 172.21.15.17:0/416269066 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2790321507 0 0) 0x7fba24148350 con 0x7fba240b4ba0 2024-03-29T11:19:06.551 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.565+0000 7fba15ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:06.551 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.565+0000 7fba15ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:06.551 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.565+0000 7fba15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:06.551 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.565+0000 7fba15ffb640 10 monclient(hunting): my global_id is 4246 2024-03-29T11:19:06.551 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.565+0000 7fba15ffb640 1 -- 172.21.15.17:0/416269066 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba08004eb0 con 0x7fba240b4ba0 2024-03-29T11:19:06.551 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.565+0000 7fba15ffb640 1 -- 172.21.15.17:0/416269066 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 654642672 0 0) 0x7fba24148630 con 0x7fba240b2520 2024-03-29T11:19:06.551 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.565+0000 7fba15ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:06.551 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.565+0000 7fba15ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:06.552 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.565+0000 7fba15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:06.552 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.565+0000 7fba15ffb640 10 monclient(hunting): my global_id is 4241 2024-03-29T11:19:06.552 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.565+0000 7fba15ffb640 1 -- 172.21.15.17:0/416269066 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba24148350 con 0x7fba240b2520 2024-03-29T11:19:06.552 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.566+0000 7fba15ffb640 1 -- 172.21.15.17:0/416269066 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3681156459 0 0) 0x7fba24148070 con 0x7fba2414fb30 2024-03-29T11:19:06.552 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.566+0000 7fba15ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:06.552 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.566+0000 7fba15ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:06.552 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.566+0000 7fba15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:06.552 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.566+0000 7fba15ffb640 10 monclient(hunting): my global_id is 4239 2024-03-29T11:19:06.552 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.566+0000 7fba15ffb640 1 -- 172.21.15.17:0/416269066 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba24148630 con 0x7fba2414fb30 2024-03-29T11:19:06.552 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.566+0000 7fba15ffb640 1 -- 172.21.15.17:0/416269066 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1877663007 0 0) 0x7fba24148350 con 0x7fba240b2520 2024-03-29T11:19:06.552 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.566+0000 7fba15ffb640 1 -- 172.21.15.17:0/416269066 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba24148070 con 0x7fba240b2520 2024-03-29T11:19:06.552 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.566+0000 7fba15ffb640 1 -- 172.21.15.17:0/416269066 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 213260026 0 0) 0x7fba08004eb0 con 0x7fba240b4ba0 2024-03-29T11:19:06.552 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.566+0000 7fba15ffb640 1 -- 172.21.15.17:0/416269066 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba24148350 con 0x7fba240b4ba0 2024-03-29T11:19:06.552 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.566+0000 7fba15ffb640 1 -- 172.21.15.17:0/416269066 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fba18001550 con 0x7fba240b2520 2024-03-29T11:19:06.553 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.566+0000 7fba15ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:06.553 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.566+0000 7fba15ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:06.553 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.566+0000 7fba15ffb640 10 monclient(hunting): dump: 2024-03-29T11:19:06.553 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:19:06.553 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:19:06.553 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:06.553 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:06.553 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:06.553 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:06.553 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:19:06.553 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:19:06.553 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:19:06.553 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:06.553 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.566+0000 7fba15ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:06.554 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.566+0000 7fba15ffb640 1 -- 172.21.15.17:0/416269066 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fba200018b0 con 0x7fba240b4ba0 2024-03-29T11:19:06.554 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.566+0000 7fba15ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:06.554 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.566+0000 7fba15ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:06.554 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.566+0000 7fba15ffb640 10 monclient(hunting): dump: 2024-03-29T11:19:06.554 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:19:06.554 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:19:06.554 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:06.554 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:06.554 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:06.554 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:06.554 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:19:06.554 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:19:06.554 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:19:06.554 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:06.555 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.566+0000 7fba15ffb640 1 -- 172.21.15.17:0/416269066 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1848471279 0 0) 0x7fba24148630 con 0x7fba2414fb30 2024-03-29T11:19:06.555 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.566+0000 7fba15ffb640 1 -- 172.21.15.17:0/416269066 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba08004eb0 con 0x7fba2414fb30 2024-03-29T11:19:06.555 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.566+0000 7fba15ffb640 1 -- 172.21.15.17:0/416269066 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fba1c001a60 con 0x7fba2414fb30 2024-03-29T11:19:06.555 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.566+0000 7fba15ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:06.555 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.566+0000 7fba15ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:06.555 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.566+0000 7fba15ffb640 10 monclient(hunting): dump: 2024-03-29T11:19:06.555 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:19:06.555 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:19:06.555 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:06.555 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:06.555 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:06.555 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:06.555 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:19:06.555 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:19:06.555 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:19:06.556 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:06.556 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.566+0000 7fba15ffb640 1 -- 172.21.15.17:0/416269066 <== mon.2 v1:172.21.15.17:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3491337656 0 0) 0x7fba24148070 con 0x7fba240b2520 2024-03-29T11:19:06.556 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.566+0000 7fba15ffb640 1 -- 172.21.15.17:0/416269066 >> v1:172.21.15.94:6789/0 conn(0x7fba240b4ba0 legacy=0x7fba2413cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:06.556 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.566+0000 7fba15ffb640 1 -- 172.21.15.17:0/416269066 >> v1:172.21.15.17:6789/0 conn(0x7fba2414fb30 legacy=0x7fba241e6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:06.556 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.566+0000 7fba15ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:19:06.556 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.566+0000 7fba15ffb640 1 monclient: found mon.c 2024-03-29T11:19:06.556 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.566+0000 7fba15ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:06.556 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.566+0000 7fba15ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:19:06.556 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.566+0000 7fba15ffb640 1 -- 172.21.15.17:0/416269066 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba24139350 con 0x7fba240b2520 2024-03-29T11:19:06.556 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.566+0000 7fba15ffb640 10 monclient: _finish_auth 0 2024-03-29T11:19:06.556 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.566+0000 7fba15ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:19:06.556 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.566+0000 7fba15ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:06.556 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.567+0000 7fba15ffb640 1 -- 172.21.15.17:0/416269066 <== mon.2 v1:172.21.15.17:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba18004a90 con 0x7fba240b2520 2024-03-29T11:19:06.556 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.567+0000 7fba15ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:06.556 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.567+0000 7fba15ffb640 1 -- 172.21.15.17:0/416269066 <== mon.2 v1:172.21.15.17:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fba18004da0 con 0x7fba240b2520 2024-03-29T11:19:06.557 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.567+0000 7fba15ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:06.557 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.567+0000 7fba15ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:06.557 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.567+0000 7fba15ffb640 10 monclient: dump: 2024-03-29T11:19:06.557 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:19:06.557 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:19:06.557 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:06.557 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:06.557 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:06.557 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:06.557 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:19:06.557 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:19:06.557 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:19:06.557 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:06.557 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.567+0000 7fba16ffd640 4 set_mon_vals no callback set 2024-03-29T11:19:06.557 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.567+0000 7fba2a02b640 5 monclient: authenticate success, global_id 4241 2024-03-29T11:19:06.558 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.567+0000 7fba2a02b640 10 monclient: _renew_subs 2024-03-29T11:19:06.558 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.567+0000 7fba2a02b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:19:06.558 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.567+0000 7fba2a02b640 1 -- 172.21.15.17:0/416269066 --> v1:172.21.15.17:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba2413d280 con 0x7fba240b2520 2024-03-29T11:19:06.558 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.567+0000 7fba2a02b640 10 client.4241.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:06.558 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.567+0000 7fba2a02b640 10 monclient: _renew_subs 2024-03-29T11:19:06.558 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.567+0000 7fba2a02b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:19:06.558 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.567+0000 7fba2a02b640 1 -- 172.21.15.17:0/416269066 --> v1:172.21.15.17:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fba241e7840 con 0x7fba240b2520 2024-03-29T11:19:06.558 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.567+0000 7fba2a02b640 1 librados: init done 2024-03-29T11:19:06.558 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.569+0000 7fba2a02b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:06.558 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.569+0000 7fba2a02b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:06.558 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.569+0000 7fba2a02b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:19:06.558 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.569+0000 7fba2a02b640 1 -- 172.21.15.17:0/416269066 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba2404f730 con 0x7fba240b2520 2024-03-29T11:19:06.560 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.570+0000 7fba15ffb640 1 -- 172.21.15.17:0/416269066 <== mon.2 v1:172.21.15.17:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7fba1801ad70 con 0x7fba240b2520 2024-03-29T11:19:06.560 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.571+0000 7fba15ffb640 1 -- 172.21.15.17:0/416269066 <== mon.2 v1:172.21.15.17:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7fba18084a70 con 0x7fba240b2520 2024-03-29T11:19:06.560 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.571+0000 7fba15ffb640 10 client.4241.objecter ms_dispatch 0x7fba24000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:06.560 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.571+0000 7fba15ffb640 3 client.4241.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:06.560 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.571+0000 7fba15ffb640 3 client.4241.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:06.560 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.571+0000 7fba28828640 10 monclient: get_auth_request con 0x7fba0806e140 auth_method 0 2024-03-29T11:19:06.560 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.571+0000 7fba15ffb640 20 client.4241.objecter dump_active .. 0 homeless 2024-03-29T11:19:06.560 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.572+0000 7fba15ffb640 10 client.4241.objecter ms_handle_connect 0x7fba0806e140 2024-03-29T11:19:06.560 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.580+0000 7fba15ffb640 1 -- 172.21.15.17:0/416269066 <== mon.2 v1:172.21.15.17:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fba18050350 con 0x7fba240b2520 2024-03-29T11:19:06.560 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.580+0000 7fba15ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:06.560 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:06.580+0000 7fba15ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:06.990 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:07.013+0000 7fba2a02b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:06.990 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:07.013+0000 7fba2a02b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:06.990 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:07.013+0000 7fba2a02b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:19:06.990 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:07.013+0000 7fba2a02b640 1 -- 172.21.15.17:0/416269066 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fba241394e0 con 0x7fba240b2520 2024-03-29T11:19:06.992 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:07.015+0000 7fba15ffb640 1 -- 172.21.15.17:0/416269066 <== mon.2 v1:172.21.15.17:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7fba241394e0 con 0x7fba240b2520 2024-03-29T11:19:06.992 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:07.015+0000 7fba15ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:06.992 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:07.015+0000 7fba15ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:06.993 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:19:06.993 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:06.997 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:07.020+0000 7fba2a02b640 10 librados: watch_flush enter 2024-03-29T11:19:06.997 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:07.020+0000 7fba2a02b640 10 librados: watch_flush exit 2024-03-29T11:19:06.997 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:07.020+0000 7fba2a02b640 20 client.4241.objecter shutdown clearing up homeless session... 2024-03-29T11:19:06.997 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:07.020+0000 7fba2a02b640 10 client.4241.objecter successfully canceled tick 2024-03-29T11:19:06.997 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:07.020+0000 7fba2a02b640 1 -- 172.21.15.17:0/416269066 >> v1:172.21.15.17:6816/3236185052 conn(0x7fba0806e140 legacy=0x7fba08070600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:06.998 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:07.020+0000 7fba2a02b640 10 monclient: shutdown 2024-03-29T11:19:06.998 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:07.020+0000 7fba2a02b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:06.998 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:07.020+0000 7fba2a02b640 1 -- 172.21.15.17:0/416269066 >> v1:172.21.15.17:6790/0 conn(0x7fba240b2520 legacy=0x7fba2413ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:06.998 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:07.020+0000 7fba2a02b640 1 -- 172.21.15.17:0/416269066 shutdown_connections 2024-03-29T11:19:06.998 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:07.020+0000 7fba2a02b640 1 -- 172.21.15.17:0/416269066 >> 172.21.15.17:0/416269066 conn(0x7fba24066be0 msgr2=0x7fba24069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:06.998 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:07.020+0000 7fba2a02b640 1 -- 172.21.15.17:0/416269066 shutdown_connections 2024-03-29T11:19:06.998 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:07.020+0000 7fba2a02b640 1 -- 172.21.15.17:0/416269066 wait complete. 2024-03-29T11:19:06.998 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:07.020+0000 7fba2a02b640 1 librados: shutdown 2024-03-29T11:19:07.011 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:13.818 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:13.953 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.976+0000 7f27b224e640 10 monclient: get_monmap_and_config 2024-03-29T11:19:13.954 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.976+0000 7f27b224e640 10 monclient: build_initial_monmap 2024-03-29T11:19:13.954 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.976+0000 7f27b224e640 10 monclient: monmap: 2024-03-29T11:19:13.954 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:19:13.954 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:19:13.954 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:19:13.977800+0000 2024-03-29T11:19:13.954 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:19:13.977800+0000 2024-03-29T11:19:13.954 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:13.954 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:13.954 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:19:13.954 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:19:13.954 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:19:13.954 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:13.956 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.977+0000 7f27b224e640 1 Processor -- start 2024-03-29T11:19:13.956 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.977+0000 7f27b224e640 1 -- start start 2024-03-29T11:19:13.956 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.977+0000 7f27b224e640 10 monclient: init 2024-03-29T11:19:13.956 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.978+0000 7f27b224e640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:13.956 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.979+0000 7f27b224e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:13.956 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.979+0000 7f27b224e640 10 monclient(hunting): picked mon.noname-a con 0x7f27ac1457f0 addr v1:172.21.15.17:6789/0 2024-03-29T11:19:13.956 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.979+0000 7f27b224e640 10 monclient(hunting): picked mon.noname-b con 0x7f27ac13f650 addr v1:172.21.15.17:6790/0 2024-03-29T11:19:13.956 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.979+0000 7f27b224e640 10 monclient(hunting): picked mon.noname-c con 0x7f27ac0a9d70 addr v1:172.21.15.94:6789/0 2024-03-29T11:19:13.956 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.979+0000 7f27b224e640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27ac0aaef0 con 0x7f27ac1457f0 2024-03-29T11:19:13.957 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.979+0000 7f27b224e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27ac145bd0 con 0x7f27ac0a9d70 2024-03-29T11:19:13.957 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.979+0000 7f27b224e640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27ac145eb0 con 0x7f27ac13f650 2024-03-29T11:19:13.957 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.979+0000 7f27b224e640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:13.957 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.979+0000 7f27b224e640 10 monclient(hunting): authenticate will time out at 860.184008s 2024-03-29T11:19:13.957 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.979+0000 7f27abfff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f27ac0a9d70 0x7f27ac14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:44126/0 (socket says 172.21.15.17:44126) 2024-03-29T11:19:13.957 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.979+0000 7f27abfff640 1 -- 172.21.15.17:0/342944667 learned_addr learned my addr 172.21.15.17:0/342944667 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:19:13.958 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.980+0000 7f27aa7fc640 1 -- 172.21.15.17:0/342944667 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 966533063 0 0) 0x7f27ac145eb0 con 0x7f27ac13f650 2024-03-29T11:19:13.958 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.980+0000 7f27aa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:13.958 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.980+0000 7f27aa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:13.958 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.980+0000 7f27aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:13.958 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.980+0000 7f27aa7fc640 10 monclient(hunting): my global_id is 4244 2024-03-29T11:19:13.958 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.980+0000 7f27aa7fc640 1 -- 172.21.15.17:0/342944667 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2794003520 con 0x7f27ac13f650 2024-03-29T11:19:13.958 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.980+0000 7f27aa7fc640 1 -- 172.21.15.17:0/342944667 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3838187539 0 0) 0x7f27ac145bd0 con 0x7f27ac0a9d70 2024-03-29T11:19:13.958 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.980+0000 7f27aa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:13.958 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.980+0000 7f27aa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:13.959 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.980+0000 7f27aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:13.959 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.980+0000 7f27aa7fc640 10 monclient(hunting): my global_id is 4249 2024-03-29T11:19:13.959 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.980+0000 7f27aa7fc640 1 -- 172.21.15.17:0/342944667 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27ac145eb0 con 0x7f27ac0a9d70 2024-03-29T11:19:13.959 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.980+0000 7f27aa7fc640 1 -- 172.21.15.17:0/342944667 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3672577106 0 0) 0x7f2794003520 con 0x7f27ac13f650 2024-03-29T11:19:13.959 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.980+0000 7f27aa7fc640 1 -- 172.21.15.17:0/342944667 >> v1:172.21.15.94:6789/0 conn(0x7f27ac0a9d70 legacy=0x7f27ac14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:13.959 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.980+0000 7f27aa7fc640 1 -- 172.21.15.17:0/342944667 >> v1:172.21.15.17:6789/0 conn(0x7f27ac1457f0 legacy=0x7f27ac13ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:19:13.960 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.982+0000 7f27aa7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:19:13.960 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.982+0000 7f27aa7fc640 1 monclient: found mon.noname-b 2024-03-29T11:19:13.961 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.982+0000 7f27aa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:13.961 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.982+0000 7f27aa7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.17:6790/0 2024-03-29T11:19:13.961 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.982+0000 7f27aa7fc640 1 -- 172.21.15.17:0/342944667 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27ac146340 con 0x7f27ac13f650 2024-03-29T11:19:13.961 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.982+0000 7f27aa7fc640 10 monclient: _finish_auth 0 2024-03-29T11:19:13.961 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.982+0000 7f27aa7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:19:13.961 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.982+0000 7f27aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:13.961 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.982+0000 7f27aa7fc640 1 -- 172.21.15.17:0/342944667 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f27a0001690 con 0x7f27ac13f650 2024-03-29T11:19:13.961 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.982+0000 7f27aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:13.961 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.982+0000 7f27aa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:13.961 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.982+0000 7f27aa7fc640 10 monclient: dump: 2024-03-29T11:19:13.961 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:19:13.961 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:19:13.961 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:13.961 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:13.962 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:13.962 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:13.962 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:19:13.962 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:19:13.962 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:19:13.962 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:13.962 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.982+0000 7f27b224e640 5 monclient: authenticate success, global_id 4244 2024-03-29T11:19:13.962 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.982+0000 7f27b224e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:13.962 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.983+0000 7f27aa7fc640 1 -- 172.21.15.17:0/342944667 <== mon.2 v1:172.21.15.17:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27a0001870 con 0x7f27ac13f650 2024-03-29T11:19:13.962 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.983+0000 7f27aa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:13.962 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.983+0000 7f27aa7fc640 1 -- 172.21.15.17:0/342944667 <== mon.2 v1:172.21.15.17:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f27a0001c80 con 0x7f27ac13f650 2024-03-29T11:19:13.962 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.983+0000 7f27aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:13.962 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.983+0000 7f27aa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:13.962 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.983+0000 7f27aa7fc640 10 monclient: dump: 2024-03-29T11:19:13.963 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:19:13.963 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:19:13.963 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:13.963 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:13.963 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:13.963 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:13.963 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:19:13.963 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:19:13.963 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:19:13.963 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:13.964 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.984+0000 7f27b224e640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:13.964 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.984+0000 7f27b224e640 4 set_mon_vals no callback set 2024-03-29T11:19:13.966 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.985+0000 7f27b224e640 10 monclient: shutdown 2024-03-29T11:19:13.966 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.985+0000 7f27b224e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:13.966 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.985+0000 7f27b224e640 1 -- 172.21.15.17:0/342944667 >> v1:172.21.15.17:6790/0 conn(0x7f27ac13f650 legacy=0x7f27ac0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:13.966 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.985+0000 7f27b224e640 1 -- 172.21.15.17:0/342944667 shutdown_connections 2024-03-29T11:19:13.966 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.985+0000 7f27b224e640 1 -- 172.21.15.17:0/342944667 >> 172.21.15.17:0/342944667 conn(0x7f27ac066a00 msgr2=0x7f27ac066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:13.966 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.985+0000 7f27b224e640 1 -- 172.21.15.17:0/342944667 shutdown_connections 2024-03-29T11:19:13.967 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.985+0000 7f27b224e640 1 -- 172.21.15.17:0/342944667 wait complete. 2024-03-29T11:19:13.967 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.986+0000 7f27b224e640 10 monclient: build_initial_monmap 2024-03-29T11:19:13.967 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.986+0000 7f27b224e640 10 monclient: monmap: 2024-03-29T11:19:13.967 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:19:13.967 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:13.967 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:19:13.967 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:19:13.967 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:13.967 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:13.967 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:19:13.967 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:19:13.967 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:19:13.968 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:13.968 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.987+0000 7f27b224e640 1 librados: starting msgr at 2024-03-29T11:19:13.968 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.987+0000 7f27b224e640 1 librados: starting objecter 2024-03-29T11:19:13.968 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.987+0000 7f27b224e640 1 Processor -- start 2024-03-29T11:19:13.968 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.987+0000 7f27b224e640 1 -- start start 2024-03-29T11:19:13.968 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.988+0000 7f27b224e640 1 librados: setting wanted keys 2024-03-29T11:19:13.968 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.988+0000 7f27b224e640 1 librados: calling monclient init 2024-03-29T11:19:13.968 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.988+0000 7f27b224e640 10 monclient: init 2024-03-29T11:19:13.968 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.989+0000 7f27b224e640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:13.968 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.989+0000 7f27b224e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:13.968 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.989+0000 7f27b224e640 10 monclient(hunting): picked mon.noname-c con 0x7f27ac0a9d70 addr v1:172.21.15.17:6790/0 2024-03-29T11:19:13.968 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.989+0000 7f27ab7fe640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f27ac0a9d70 0x7f27ac1425f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:38062/0 (socket says 172.21.15.17:38062) 2024-03-29T11:19:13.968 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.989+0000 7f27ab7fe640 1 -- 172.21.15.17:0/1626271845 learned_addr learned my addr 172.21.15.17:0/1626271845 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:19:13.968 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.989+0000 7f27b224e640 10 monclient(hunting): picked mon.noname-b con 0x7f27ac13f650 addr v1:172.21.15.94:6789/0 2024-03-29T11:19:13.969 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.989+0000 7f2793fff640 10 client.?.objecter ms_handle_connect 0x7f27ac0a9d70 2024-03-29T11:19:13.971 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.989+0000 7f2793fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:13.971 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.990+0000 7f27b224e640 10 monclient(hunting): picked mon.noname-a con 0x7f27ac1457f0 addr v1:172.21.15.17:6789/0 2024-03-29T11:19:13.971 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.990+0000 7f27b224e640 1 -- 172.21.15.17:0/1626271845 --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27ac141e30 con 0x7f27ac1457f0 2024-03-29T11:19:13.971 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.990+0000 7f27b224e640 1 -- 172.21.15.17:0/1626271845 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27ac06f0f0 con 0x7f27ac13f650 2024-03-29T11:19:13.971 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.990+0000 7f27b224e640 1 -- 172.21.15.17:0/1626271845 --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27ac06e250 con 0x7f27ac0a9d70 2024-03-29T11:19:13.972 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.990+0000 7f27b224e640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:13.972 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.990+0000 7f27b224e640 10 monclient(hunting): authenticate will time out at 860.195388s 2024-03-29T11:19:13.972 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.990+0000 7f2793fff640 10 client.?.objecter ms_handle_connect 0x7f27ac13f650 2024-03-29T11:19:13.972 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.990+0000 7f2793fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:13.972 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.990+0000 7f2793fff640 1 -- 172.21.15.17:0/1626271845 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2891775603 0 0) 0x7f27ac06f0f0 con 0x7f27ac13f650 2024-03-29T11:19:13.972 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.990+0000 7f2793fff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:13.972 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.990+0000 7f2793fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:13.972 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.990+0000 7f2793fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:13.972 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.990+0000 7f2793fff640 10 monclient(hunting): my global_id is 4252 2024-03-29T11:19:13.972 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.991+0000 7f2793fff640 1 -- 172.21.15.17:0/1626271845 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2798004eb0 con 0x7f27ac13f650 2024-03-29T11:19:13.972 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.991+0000 7f2793fff640 1 -- 172.21.15.17:0/1626271845 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1772356871 0 0) 0x7f2798004eb0 con 0x7f27ac13f650 2024-03-29T11:19:13.972 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.991+0000 7f2793fff640 1 -- 172.21.15.17:0/1626271845 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27ac06f0f0 con 0x7f27ac13f650 2024-03-29T11:19:13.972 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.991+0000 7f2793fff640 1 -- 172.21.15.17:0/1626271845 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f27a0001bc0 con 0x7f27ac13f650 2024-03-29T11:19:13.972 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.991+0000 7f2793fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:13.973 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.991+0000 7f2793fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:13.973 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.991+0000 7f2793fff640 10 monclient(hunting): dump: 2024-03-29T11:19:13.973 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:19:13.973 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:19:13.973 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:13.973 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:13.973 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:13.973 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:13.973 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:19:13.973 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:19:13.973 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:19:13.973 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:13.973 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.991+0000 7f2793fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:13.973 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.991+0000 7f2793fff640 1 -- 172.21.15.17:0/1626271845 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 352928346 0 0) 0x7f27ac06e250 con 0x7f27ac0a9d70 2024-03-29T11:19:13.973 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.991+0000 7f2793fff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:13.974 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.991+0000 7f2793fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:13.974 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.991+0000 7f2793fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:13.974 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.991+0000 7f2793fff640 10 monclient(hunting): my global_id is 4247 2024-03-29T11:19:13.974 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.991+0000 7f2793fff640 1 -- 172.21.15.17:0/1626271845 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2798004eb0 con 0x7f27ac0a9d70 2024-03-29T11:19:13.974 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.991+0000 7f2793fff640 1 -- 172.21.15.17:0/1626271845 <== mon.1 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2965901916 0 0) 0x7f27ac06f0f0 con 0x7f27ac13f650 2024-03-29T11:19:13.974 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.991+0000 7f2793fff640 1 -- 172.21.15.17:0/1626271845 >> v1:172.21.15.17:6790/0 conn(0x7f27ac0a9d70 legacy=0x7f27ac1425f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:13.974 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.992+0000 7f2793fff640 1 -- 172.21.15.17:0/1626271845 >> v1:172.21.15.17:6789/0 conn(0x7f27ac1457f0 legacy=0x7f27ac1e7070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:19:13.974 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.992+0000 7f2793fff640 10 monclient: _finish_hunting 0 2024-03-29T11:19:13.974 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.992+0000 7f2793fff640 1 monclient: found mon.b 2024-03-29T11:19:13.974 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.992+0000 7f2793fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:13.974 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.992+0000 7f2793fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:19:13.974 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.992+0000 7f2793fff640 1 -- 172.21.15.17:0/1626271845 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27ac06e3e0 con 0x7f27ac13f650 2024-03-29T11:19:13.974 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.992+0000 7f2793fff640 10 monclient: _finish_auth 0 2024-03-29T11:19:13.975 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.992+0000 7f2793fff640 10 monclient: _check_auth_tickets 2024-03-29T11:19:13.975 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.993+0000 7f2793fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:13.975 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.993+0000 7f27b224e640 5 monclient: authenticate success, global_id 4252 2024-03-29T11:19:13.975 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.993+0000 7f27b224e640 10 monclient: _renew_subs 2024-03-29T11:19:13.975 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.993+0000 7f27b224e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:19:13.975 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.993+0000 7f27b224e640 1 -- 172.21.15.17:0/1626271845 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27ac06de90 con 0x7f27ac13f650 2024-03-29T11:19:13.975 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.993+0000 7f27b224e640 10 client.4252.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:13.975 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.993+0000 7f2793fff640 1 -- 172.21.15.17:0/1626271845 <== mon.1 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27a0003940 con 0x7f27ac13f650 2024-03-29T11:19:13.975 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.993+0000 7f2793fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:13.975 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.993+0000 7f27b224e640 10 monclient: _renew_subs 2024-03-29T11:19:13.975 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.993+0000 7f2793fff640 1 -- 172.21.15.17:0/1626271845 <== mon.1 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f27a0003c50 con 0x7f27ac13f650 2024-03-29T11:19:13.975 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.993+0000 7f27b224e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:19:13.975 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.993+0000 7f27b224e640 1 -- 172.21.15.17:0/1626271845 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f27ac13b710 con 0x7f27ac13f650 2024-03-29T11:19:13.975 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.993+0000 7f27b224e640 1 librados: init done 2024-03-29T11:19:13.976 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.993+0000 7f2793fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:13.976 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.993+0000 7f2793fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:13.976 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.993+0000 7f2793fff640 10 monclient: dump: 2024-03-29T11:19:13.976 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:19:13.976 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:19:13.976 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:13.976 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:13.976 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:13.976 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:13.976 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:19:13.976 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:19:13.976 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:19:13.976 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:13.976 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.993+0000 7f27a9ffb640 4 set_mon_vals no callback set 2024-03-29T11:19:13.978 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.995+0000 7f2793fff640 1 -- 172.21.15.17:0/1626271845 <== mon.1 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f27a0019c20 con 0x7f27ac13f650 2024-03-29T11:19:13.978 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.996+0000 7f27b224e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:13.978 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.996+0000 7f27b224e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:13.978 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.996+0000 7f27b224e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:19:13.978 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.996+0000 7f27b224e640 1 -- 172.21.15.17:0/1626271845 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f278c005080 con 0x7f27ac13f650 2024-03-29T11:19:13.978 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.996+0000 7f27ab7fe640 10 monclient: get_auth_request con 0x7f279806dd00 auth_method 0 2024-03-29T11:19:13.978 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.996+0000 7f2793fff640 1 -- 172.21.15.17:0/1626271845 <== mon.1 v1:172.21.15.94:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f27a0083a10 con 0x7f27ac13f650 2024-03-29T11:19:13.978 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.996+0000 7f2793fff640 10 client.4252.objecter ms_dispatch 0x7f27ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:13.978 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.996+0000 7f2793fff640 3 client.4252.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:13.978 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.996+0000 7f2793fff640 3 client.4252.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:13.978 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.996+0000 7f2793fff640 20 client.4252.objecter dump_active .. 0 homeless 2024-03-29T11:19:13.978 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:13.996+0000 7f2793fff640 10 client.4252.objecter ms_handle_connect 0x7f279806dd00 2024-03-29T11:19:13.981 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:14.004+0000 7f2793fff640 1 -- 172.21.15.17:0/1626271845 <== mon.1 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f27a004f2f0 con 0x7f27ac13f650 2024-03-29T11:19:13.981 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:14.004+0000 7f2793fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:13.981 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:14.004+0000 7f2793fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:14.308 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:14.331+0000 7f27b224e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:14.308 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:14.331+0000 7f27b224e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:14.309 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:14.331+0000 7f27b224e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:19:14.309 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:14.331+0000 7f27b224e640 1 -- 172.21.15.17:0/1626271845 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f278c005330 con 0x7f27ac13f650 2024-03-29T11:19:14.312 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:14.335+0000 7f2793fff640 1 -- 172.21.15.17:0/1626271845 <== mon.1 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f27a0051220 con 0x7f27ac13f650 2024-03-29T11:19:14.312 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:14.335+0000 7f2793fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:14.312 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:14.335+0000 7f2793fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:14.312 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:19:14.312 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:14.315 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:14.338+0000 7f27b224e640 10 librados: watch_flush enter 2024-03-29T11:19:14.315 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:14.338+0000 7f27b224e640 10 librados: watch_flush exit 2024-03-29T11:19:14.315 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:14.338+0000 7f27b224e640 20 client.4252.objecter shutdown clearing up homeless session... 2024-03-29T11:19:14.315 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:14.338+0000 7f27b224e640 10 client.4252.objecter successfully canceled tick 2024-03-29T11:19:14.316 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:14.338+0000 7f27b224e640 1 -- 172.21.15.17:0/1626271845 >> v1:172.21.15.17:6816/3236185052 conn(0x7f279806dd00 legacy=0x7f27980701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:14.316 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:14.339+0000 7f27b224e640 10 monclient: shutdown 2024-03-29T11:19:14.316 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:14.339+0000 7f27b224e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:14.316 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:14.339+0000 7f27b224e640 1 -- 172.21.15.17:0/1626271845 >> v1:172.21.15.94:6789/0 conn(0x7f27ac13f650 legacy=0x7f27ac137450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:14.316 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:14.339+0000 7f27b224e640 1 -- 172.21.15.17:0/1626271845 shutdown_connections 2024-03-29T11:19:14.316 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:14.339+0000 7f27b224e640 1 -- 172.21.15.17:0/1626271845 >> 172.21.15.17:0/1626271845 conn(0x7f27ac066a00 msgr2=0x7f27ac0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:14.316 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:14.339+0000 7f27b224e640 1 -- 172.21.15.17:0/1626271845 shutdown_connections 2024-03-29T11:19:14.316 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:14.339+0000 7f27b224e640 1 -- 172.21.15.17:0/1626271845 wait complete. 2024-03-29T11:19:14.316 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:14.339+0000 7f27b224e640 1 librados: shutdown 2024-03-29T11:19:14.331 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:21.138 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:21.271 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.294+0000 7f1ffe723640 10 monclient: get_monmap_and_config 2024-03-29T11:19:21.271 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.294+0000 7f1ffe723640 10 monclient: build_initial_monmap 2024-03-29T11:19:21.271 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.294+0000 7f1ffe723640 10 monclient: monmap: 2024-03-29T11:19:21.271 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:19:21.271 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:19:21.271 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:19:21.295244+0000 2024-03-29T11:19:21.271 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:19:21.295244+0000 2024-03-29T11:19:21.271 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:21.271 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:21.271 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:19:21.272 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:19:21.272 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:19:21.272 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:21.272 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.295+0000 7f1ffe723640 1 Processor -- start 2024-03-29T11:19:21.274 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.295+0000 7f1ffe723640 1 -- start start 2024-03-29T11:19:21.274 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.295+0000 7f1ffe723640 10 monclient: init 2024-03-29T11:19:21.274 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.296+0000 7f1ffe723640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:21.274 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.296+0000 7f1ffe723640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:21.274 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.296+0000 7f1ffe723640 10 monclient(hunting): picked mon.noname-c con 0x7f1ff806f050 addr v1:172.21.15.94:6789/0 2024-03-29T11:19:21.274 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.296+0000 7f1ffe723640 10 monclient(hunting): picked mon.noname-b con 0x7f1ff8070240 addr v1:172.21.15.17:6790/0 2024-03-29T11:19:21.274 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.296+0000 7f1ffe723640 10 monclient(hunting): picked mon.noname-a con 0x7f1ff806f430 addr v1:172.21.15.17:6789/0 2024-03-29T11:19:21.274 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.296+0000 7f1ffe723640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff813f400 con 0x7f1ff806f430 2024-03-29T11:19:21.275 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.296+0000 7f1ffe723640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff81429c0 con 0x7f1ff806f050 2024-03-29T11:19:21.275 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.296+0000 7f1ffe723640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff806ff80 con 0x7f1ff8070240 2024-03-29T11:19:21.275 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.296+0000 7f1ffe723640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:21.275 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.296+0000 7f1ffe723640 10 monclient(hunting): authenticate will time out at 867.501401s 2024-03-29T11:19:21.275 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.297+0000 7f1ff77fe640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f1ff8070240 0x7f1ff80ae860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:35054/0 (socket says 172.21.15.17:35054) 2024-03-29T11:19:21.275 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.297+0000 7f1ff77fe640 1 -- 172.21.15.17:0/1642186747 learned_addr learned my addr 172.21.15.17:0/1642186747 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:19:21.275 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.297+0000 7f1ff6ffd640 1 -- 172.21.15.17:0/1642186747 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1873476290 0 0) 0x7f1ff806ff80 con 0x7f1ff8070240 2024-03-29T11:19:21.275 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.297+0000 7f1ff6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:21.275 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.297+0000 7f1ff6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:21.275 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.297+0000 7f1ff6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:21.275 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.297+0000 7f1ff6ffd640 10 monclient(hunting): my global_id is 4250 2024-03-29T11:19:21.275 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.297+0000 7f1ff6ffd640 1 -- 172.21.15.17:0/1642186747 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fe0003520 con 0x7f1ff8070240 2024-03-29T11:19:21.277 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.298+0000 7f1ff6ffd640 1 -- 172.21.15.17:0/1642186747 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 601987433 0 0) 0x7f1fe0003520 con 0x7f1ff8070240 2024-03-29T11:19:21.277 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.298+0000 7f1ff6ffd640 1 -- 172.21.15.17:0/1642186747 >> v1:172.21.15.94:6789/0 conn(0x7f1ff806f050 legacy=0x7f1ff81422b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:19:21.277 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.298+0000 7f1ff6ffd640 1 -- 172.21.15.17:0/1642186747 >> v1:172.21.15.17:6789/0 conn(0x7f1ff806f430 legacy=0x7f1ff814a630 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:19:21.277 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.299+0000 7f1ff6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:21.277 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.299+0000 7f1ff6ffd640 1 monclient: found mon.noname-b 2024-03-29T11:19:21.277 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.299+0000 7f1ff6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:21.277 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.299+0000 7f1ff6ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.17:6790/0 2024-03-29T11:19:21.278 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.299+0000 7f1ff6ffd640 1 -- 172.21.15.17:0/1642186747 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff80aef70 con 0x7f1ff8070240 2024-03-29T11:19:21.278 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.299+0000 7f1ff6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:19:21.278 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.299+0000 7f1ff6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:21.278 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.299+0000 7f1ff6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:21.278 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.299+0000 7f1ffe723640 5 monclient: authenticate success, global_id 4250 2024-03-29T11:19:21.278 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.299+0000 7f1ffe723640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:21.278 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.299+0000 7f1ff6ffd640 1 -- 172.21.15.17:0/1642186747 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f1ff0001a10 con 0x7f1ff8070240 2024-03-29T11:19:21.278 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.299+0000 7f1ff6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:21.278 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.299+0000 7f1ff6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:21.278 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.299+0000 7f1ff6ffd640 10 monclient: dump: 2024-03-29T11:19:21.278 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:19:21.278 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:19:21.278 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:21.279 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:21.279 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:21.279 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:21.279 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:19:21.279 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:19:21.279 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:19:21.279 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:21.279 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.300+0000 7f1ffe723640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:21.279 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.300+0000 7f1ff6ffd640 1 -- 172.21.15.17:0/1642186747 <== mon.2 v1:172.21.15.17:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ff0001ca0 con 0x7f1ff8070240 2024-03-29T11:19:21.279 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.300+0000 7f1ff6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:21.279 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.300+0000 7f1ffe723640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:21.279 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.300+0000 7f1ffe723640 4 set_mon_vals no callback set 2024-03-29T11:19:21.279 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.300+0000 7f1ff6ffd640 1 -- 172.21.15.17:0/1642186747 <== mon.2 v1:172.21.15.17:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f1ff0002030 con 0x7f1ff8070240 2024-03-29T11:19:21.279 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.300+0000 7f1ff6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:21.279 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.300+0000 7f1ff6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:21.280 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.300+0000 7f1ff6ffd640 10 monclient: dump: 2024-03-29T11:19:21.280 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:19:21.280 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:19:21.280 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:21.280 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:21.280 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:21.280 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:21.280 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:19:21.280 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:19:21.280 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:19:21.280 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:21.284 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.301+0000 7f1ffe723640 10 monclient: shutdown 2024-03-29T11:19:21.284 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.301+0000 7f1ffe723640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:21.284 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.301+0000 7f1ffe723640 1 -- 172.21.15.17:0/1642186747 >> v1:172.21.15.17:6790/0 conn(0x7f1ff8070240 legacy=0x7f1ff80ae860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:21.285 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.301+0000 7f1ffe723640 1 -- 172.21.15.17:0/1642186747 shutdown_connections 2024-03-29T11:19:21.285 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.301+0000 7f1ffe723640 1 -- 172.21.15.17:0/1642186747 >> 172.21.15.17:0/1642186747 conn(0x7f1ff80b4930 msgr2=0x7f1ff80b4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:21.285 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.301+0000 7f1ffe723640 1 -- 172.21.15.17:0/1642186747 shutdown_connections 2024-03-29T11:19:21.285 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.301+0000 7f1ffe723640 1 -- 172.21.15.17:0/1642186747 wait complete. 2024-03-29T11:19:21.285 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.302+0000 7f1ffe723640 10 monclient: build_initial_monmap 2024-03-29T11:19:21.285 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.302+0000 7f1ffe723640 10 monclient: monmap: 2024-03-29T11:19:21.285 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:19:21.285 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:21.285 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:19:21.285 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:19:21.285 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:21.285 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:21.285 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:19:21.286 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:19:21.286 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:19:21.286 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:21.286 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.302+0000 7f1ffe723640 1 librados: starting msgr at 2024-03-29T11:19:21.286 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.302+0000 7f1ffe723640 1 librados: starting objecter 2024-03-29T11:19:21.286 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.302+0000 7f1ffe723640 1 Processor -- start 2024-03-29T11:19:21.286 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.302+0000 7f1ffe723640 1 -- start start 2024-03-29T11:19:21.286 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.302+0000 7f1ffe723640 1 librados: setting wanted keys 2024-03-29T11:19:21.286 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.302+0000 7f1ffe723640 1 librados: calling monclient init 2024-03-29T11:19:21.286 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.302+0000 7f1ffe723640 10 monclient: init 2024-03-29T11:19:21.286 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.303+0000 7f1ffe723640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:21.286 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.303+0000 7f1ffe723640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:19:21.286 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.303+0000 7f1ffe723640 10 monclient(hunting): picked mon.noname-c con 0x7f1ff806f050 addr v1:172.21.15.17:6790/0 2024-03-29T11:19:21.286 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.304+0000 7f1ffe723640 10 monclient(hunting): picked mon.noname-a con 0x7f1ff806f430 addr v1:172.21.15.17:6789/0 2024-03-29T11:19:21.286 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.304+0000 7f1ff7fff640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f1ff806f050 0x7f1ff80a9750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:35966/0 (socket says 172.21.15.17:35966) 2024-03-29T11:19:21.287 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.304+0000 7f1ff7fff640 1 -- 172.21.15.17:0/3730756099 learned_addr learned my addr 172.21.15.17:0/3730756099 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:19:21.287 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.304+0000 7f1ffe723640 10 monclient(hunting): picked mon.noname-b con 0x7f1ff8070240 addr v1:172.21.15.94:6789/0 2024-03-29T11:19:21.287 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.304+0000 7f1ff4ff9640 10 client.?.objecter ms_handle_connect 0x7f1ff806f050 2024-03-29T11:19:21.287 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.304+0000 7f1ff4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:21.287 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.304+0000 7f1ffe723640 1 -- 172.21.15.17:0/3730756099 --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff813cb70 con 0x7f1ff806f430 2024-03-29T11:19:21.287 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.304+0000 7f1ffe723640 1 -- 172.21.15.17:0/3730756099 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff813ee70 con 0x7f1ff8070240 2024-03-29T11:19:21.287 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.304+0000 7f1ffe723640 1 -- 172.21.15.17:0/3730756099 --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff80a9f70 con 0x7f1ff806f050 2024-03-29T11:19:21.287 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.304+0000 7f1ffe723640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:21.287 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.304+0000 7f1ffe723640 10 monclient(hunting): authenticate will time out at 867.509510s 2024-03-29T11:19:21.287 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.304+0000 7f1ff4ff9640 1 -- 172.21.15.17:0/3730756099 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2933484335 0 0) 0x7f1ff80a9f70 con 0x7f1ff806f050 2024-03-29T11:19:21.287 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.304+0000 7f1ff4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:21.287 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.304+0000 7f1ff4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:21.287 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.304+0000 7f1ff4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:21.288 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.304+0000 7f1ff4ff9640 10 monclient(hunting): my global_id is 4253 2024-03-29T11:19:21.288 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.304+0000 7f1ff4ff9640 1 -- 172.21.15.17:0/3730756099 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fe4004eb0 con 0x7f1ff806f050 2024-03-29T11:19:21.288 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.305+0000 7f1ff4ff9640 1 -- 172.21.15.17:0/3730756099 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3039265117 0 0) 0x7f1fe4004eb0 con 0x7f1ff806f050 2024-03-29T11:19:21.288 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.305+0000 7f1ff4ff9640 1 -- 172.21.15.17:0/3730756099 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ff80a9f70 con 0x7f1ff806f050 2024-03-29T11:19:21.288 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.305+0000 7f1ff4ff9640 1 -- 172.21.15.17:0/3730756099 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f1fec001960 con 0x7f1ff806f050 2024-03-29T11:19:21.288 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.305+0000 7f1ff4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:21.288 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.305+0000 7f1ff4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:21.288 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.305+0000 7f1ff4ff9640 10 monclient(hunting): dump: 2024-03-29T11:19:21.288 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:19:21.288 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:19:21.288 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:21.288 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:21.288 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:21.288 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:21.289 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:19:21.289 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:19:21.289 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:19:21.289 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:21.289 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.305+0000 7f1ff4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:21.289 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.305+0000 7f1ff4ff9640 10 client.?.objecter ms_handle_connect 0x7f1ff8070240 2024-03-29T11:19:21.289 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.305+0000 7f1ff4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:21.289 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.305+0000 7f1ff4ff9640 1 -- 172.21.15.17:0/3730756099 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 519296084 0 0) 0x7f1ff813ee70 con 0x7f1ff8070240 2024-03-29T11:19:21.289 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.305+0000 7f1ff4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:21.289 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.305+0000 7f1ff4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:21.289 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.305+0000 7f1ff4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:21.289 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.305+0000 7f1ff4ff9640 10 monclient(hunting): my global_id is 4255 2024-03-29T11:19:21.289 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.305+0000 7f1ff4ff9640 1 -- 172.21.15.17:0/3730756099 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fe4004eb0 con 0x7f1ff8070240 2024-03-29T11:19:21.289 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.305+0000 7f1ff4ff9640 1 -- 172.21.15.17:0/3730756099 <== mon.2 v1:172.21.15.17:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 814181767 0 0) 0x7f1ff80a9f70 con 0x7f1ff806f050 2024-03-29T11:19:21.290 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.305+0000 7f1ff4ff9640 1 -- 172.21.15.17:0/3730756099 >> v1:172.21.15.94:6789/0 conn(0x7f1ff8070240 legacy=0x7f1ff813da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:21.290 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.305+0000 7f1ff4ff9640 1 -- 172.21.15.17:0/3730756099 >> v1:172.21.15.17:6789/0 conn(0x7f1ff806f430 legacy=0x7f1ff813d330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:19:21.290 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.307+0000 7f1ff4ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:19:21.290 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.307+0000 7f1ff4ff9640 1 monclient: found mon.c 2024-03-29T11:19:21.290 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.307+0000 7f1ff4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:21.290 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.307+0000 7f1ff4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:19:21.290 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.307+0000 7f1ff4ff9640 1 -- 172.21.15.17:0/3730756099 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff80aa100 con 0x7f1ff806f050 2024-03-29T11:19:21.290 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.307+0000 7f1ff4ff9640 10 monclient: _finish_auth 0 2024-03-29T11:19:21.290 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.307+0000 7f1ff4ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:19:21.290 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.307+0000 7f1ff4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:21.290 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.307+0000 7f1ffe723640 5 monclient: authenticate success, global_id 4253 2024-03-29T11:19:21.290 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.307+0000 7f1ffe723640 10 monclient: _renew_subs 2024-03-29T11:19:21.290 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.307+0000 7f1ffe723640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:19:21.290 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.307+0000 7f1ffe723640 1 -- 172.21.15.17:0/3730756099 --> v1:172.21.15.17:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ff81e2b30 con 0x7f1ff806f050 2024-03-29T11:19:21.291 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.307+0000 7f1ffe723640 10 client.4253.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:21.291 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.307+0000 7f1ffe723640 10 monclient: _renew_subs 2024-03-29T11:19:21.291 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.307+0000 7f1ffe723640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:19:21.291 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.307+0000 7f1ffe723640 1 -- 172.21.15.17:0/3730756099 --> v1:172.21.15.17:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ff81e2ea0 con 0x7f1ff806f050 2024-03-29T11:19:21.291 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.307+0000 7f1ffe723640 1 librados: init done 2024-03-29T11:19:21.293 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.308+0000 7f1ff4ff9640 1 -- 172.21.15.17:0/3730756099 <== mon.2 v1:172.21.15.17:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1fec001ac0 con 0x7f1ff806f050 2024-03-29T11:19:21.293 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.308+0000 7f1ff4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:21.293 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.308+0000 7f1ff4ff9640 1 -- 172.21.15.17:0/3730756099 <== mon.2 v1:172.21.15.17:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f1fec001d80 con 0x7f1ff806f050 2024-03-29T11:19:21.293 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.308+0000 7f1ff4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:21.293 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.308+0000 7f1ff4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:21.293 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.308+0000 7f1ff4ff9640 10 monclient: dump: 2024-03-29T11:19:21.293 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:19:21.293 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:19:21.293 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:21.294 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:21.294 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:21.294 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:21.294 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:19:21.294 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:19:21.294 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:19:21.294 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:21.294 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.308+0000 7f1ff67fc640 4 set_mon_vals no callback set 2024-03-29T11:19:21.294 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.309+0000 7f1ff4ff9640 1 -- 172.21.15.17:0/3730756099 <== mon.2 v1:172.21.15.17:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f1fec003a40 con 0x7f1ff806f050 2024-03-29T11:19:21.294 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.310+0000 7f1ff4ff9640 1 -- 172.21.15.17:0/3730756099 <== mon.2 v1:172.21.15.17:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f1fec084ae0 con 0x7f1ff806f050 2024-03-29T11:19:21.294 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.310+0000 7f1ff4ff9640 10 client.4253.objecter ms_dispatch 0x7f1ff8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:21.294 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.310+0000 7f1ff4ff9640 3 client.4253.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:21.294 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.310+0000 7f1ff4ff9640 3 client.4253.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:21.294 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.310+0000 7f1ff4ff9640 20 client.4253.objecter dump_active .. 0 homeless 2024-03-29T11:19:21.295 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.310+0000 7f1ff77fe640 10 monclient: get_auth_request con 0x7f1fe406dd30 auth_method 0 2024-03-29T11:19:21.295 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.310+0000 7f1ffe723640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:21.295 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.310+0000 7f1ffe723640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:21.295 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.310+0000 7f1ffe723640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:19:21.295 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.310+0000 7f1ffe723640 1 -- 172.21.15.17:0/3730756099 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fd8005080 con 0x7f1ff806f050 2024-03-29T11:19:21.295 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.311+0000 7f1ff4ff9640 10 client.4253.objecter ms_handle_connect 0x7f1fe406dd30 2024-03-29T11:19:21.295 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.317+0000 7f1ff4ff9640 1 -- 172.21.15.17:0/3730756099 <== mon.2 v1:172.21.15.17:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1fec0501f0 con 0x7f1ff806f050 2024-03-29T11:19:21.295 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.318+0000 7f1ff4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:21.295 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.318+0000 7f1ff4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:21.616 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.639+0000 7f1ffe723640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:21.616 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.639+0000 7f1ffe723640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:21.616 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.639+0000 7f1ffe723640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:19:21.616 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.639+0000 7f1ffe723640 1 -- 172.21.15.17:0/3730756099 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1fd8005330 con 0x7f1ff806f050 2024-03-29T11:19:21.617 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.640+0000 7f1ff4ff9640 1 -- 172.21.15.17:0/3730756099 <== mon.2 v1:172.21.15.17:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f1fec0522f0 con 0x7f1ff806f050 2024-03-29T11:19:21.617 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.640+0000 7f1ff4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:21.617 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.640+0000 7f1ff4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:21.617 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:19:21.617 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:21.620 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.643+0000 7f1ffe723640 10 librados: watch_flush enter 2024-03-29T11:19:21.620 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.643+0000 7f1ffe723640 10 librados: watch_flush exit 2024-03-29T11:19:21.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.644+0000 7f1ffe723640 20 client.4253.objecter shutdown clearing up homeless session... 2024-03-29T11:19:21.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.644+0000 7f1ffe723640 10 client.4253.objecter successfully canceled tick 2024-03-29T11:19:21.622 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.645+0000 7f1ffe723640 1 -- 172.21.15.17:0/3730756099 >> v1:172.21.15.17:6816/3236185052 conn(0x7f1fe406dd30 legacy=0x7f1fe40701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:21.622 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.645+0000 7f1ffe723640 10 monclient: shutdown 2024-03-29T11:19:21.622 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.645+0000 7f1ffe723640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:21.622 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.645+0000 7f1ffe723640 1 -- 172.21.15.17:0/3730756099 >> v1:172.21.15.17:6790/0 conn(0x7f1ff806f050 legacy=0x7f1ff80a9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:21.622 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.645+0000 7f1ffe723640 1 -- 172.21.15.17:0/3730756099 shutdown_connections 2024-03-29T11:19:21.622 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.645+0000 7f1ffe723640 1 -- 172.21.15.17:0/3730756099 >> 172.21.15.17:0/3730756099 conn(0x7f1ff80b4930 msgr2=0x7f1ff806b740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:21.623 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.646+0000 7f1ffe723640 1 -- 172.21.15.17:0/3730756099 shutdown_connections 2024-03-29T11:19:21.623 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.646+0000 7f1ffe723640 1 -- 172.21.15.17:0/3730756099 wait complete. 2024-03-29T11:19:21.623 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:21.646+0000 7f1ffe723640 1 librados: shutdown 2024-03-29T11:19:21.639 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:28.445 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:28.592 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.614+0000 7f0f99bf4640 10 monclient: get_monmap_and_config 2024-03-29T11:19:28.592 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.614+0000 7f0f99bf4640 10 monclient: build_initial_monmap 2024-03-29T11:19:28.592 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.614+0000 7f0f99bf4640 10 monclient: monmap: 2024-03-29T11:19:28.592 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:19:28.592 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:19:28.592 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:19:28.615864+0000 2024-03-29T11:19:28.592 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:19:28.615864+0000 2024-03-29T11:19:28.592 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:28.592 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:28.592 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:19:28.593 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:19:28.593 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:19:28.593 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:28.593 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.616+0000 7f0f99bf4640 1 Processor -- start 2024-03-29T11:19:28.593 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.616+0000 7f0f99bf4640 1 -- start start 2024-03-29T11:19:28.593 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.616+0000 7f0f99bf4640 10 monclient: init 2024-03-29T11:19:28.594 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.617+0000 7f0f99bf4640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:28.594 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.617+0000 7f0f99bf4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:28.594 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.617+0000 7f0f99bf4640 10 monclient(hunting): picked mon.noname-b con 0x7f0f940b0770 addr v1:172.21.15.17:6790/0 2024-03-29T11:19:28.594 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.617+0000 7f0f99bf4640 10 monclient(hunting): picked mon.noname-c con 0x7f0f9414a700 addr v1:172.21.15.94:6789/0 2024-03-29T11:19:28.595 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.617+0000 7f0f99bf4640 10 monclient(hunting): picked mon.noname-a con 0x7f0f940ac5a0 addr v1:172.21.15.17:6789/0 2024-03-29T11:19:28.595 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.617+0000 7f0f99bf4640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f940aba70 con 0x7f0f940ac5a0 2024-03-29T11:19:28.595 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.617+0000 7f0f99bf4640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f940aecb0 con 0x7f0f9414a700 2024-03-29T11:19:28.595 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.617+0000 7f0f99bf4640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f940af150 con 0x7f0f940b0770 2024-03-29T11:19:28.595 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.617+0000 7f0f99bf4640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:28.595 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.617+0000 7f0f99bf4640 10 monclient(hunting): authenticate will time out at 874.822601s 2024-03-29T11:19:28.595 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.618+0000 7f0f92ffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f0f9414a700 0x7f0f940abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:46196/0 (socket says 172.21.15.17:46196) 2024-03-29T11:19:28.595 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.618+0000 7f0f92ffd640 1 -- 172.21.15.17:0/2437026209 learned_addr learned my addr 172.21.15.17:0/2437026209 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:19:28.596 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.619+0000 7f0f927fc640 1 -- 172.21.15.17:0/2437026209 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3934543842 0 0) 0x7f0f940aecb0 con 0x7f0f9414a700 2024-03-29T11:19:28.596 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.619+0000 7f0f927fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:28.596 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.619+0000 7f0f927fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:28.596 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.619+0000 7f0f927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:28.596 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.619+0000 7f0f927fc640 10 monclient(hunting): my global_id is 4258 2024-03-29T11:19:28.596 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.619+0000 7f0f927fc640 1 -- 172.21.15.17:0/2437026209 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f7c003520 con 0x7f0f9414a700 2024-03-29T11:19:28.600 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.621+0000 7f0f927fc640 1 -- 172.21.15.17:0/2437026209 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 316707357 0 0) 0x7f0f7c003520 con 0x7f0f9414a700 2024-03-29T11:19:28.600 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.621+0000 7f0f927fc640 1 -- 172.21.15.17:0/2437026209 >> v1:172.21.15.17:6790/0 conn(0x7f0f940b0770 legacy=0x7f0f94149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:19:28.600 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.621+0000 7f0f927fc640 1 -- 172.21.15.17:0/2437026209 >> v1:172.21.15.17:6789/0 conn(0x7f0f940ac5a0 legacy=0x7f0f94151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:19:28.600 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.621+0000 7f0f927fc640 10 monclient: _finish_hunting 0 2024-03-29T11:19:28.600 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.621+0000 7f0f927fc640 1 monclient: found mon.noname-c 2024-03-29T11:19:28.600 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.621+0000 7f0f927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:28.600 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.621+0000 7f0f927fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6789/0 2024-03-29T11:19:28.600 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.621+0000 7f0f927fc640 1 -- 172.21.15.17:0/2437026209 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f940ac9d0 con 0x7f0f9414a700 2024-03-29T11:19:28.601 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.621+0000 7f0f927fc640 10 monclient: _finish_auth 0 2024-03-29T11:19:28.601 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.621+0000 7f0f927fc640 10 monclient: _check_auth_tickets 2024-03-29T11:19:28.601 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.621+0000 7f0f927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:28.601 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.621+0000 7f0f99bf4640 5 monclient: authenticate success, global_id 4258 2024-03-29T11:19:28.601 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.621+0000 7f0f99bf4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:28.601 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.621+0000 7f0f927fc640 1 -- 172.21.15.17:0/2437026209 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f0f8c001a30 con 0x7f0f9414a700 2024-03-29T11:19:28.601 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.621+0000 7f0f927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:28.601 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.621+0000 7f0f927fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:28.601 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.621+0000 7f0f927fc640 10 monclient: dump: 2024-03-29T11:19:28.601 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:19:28.601 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:19:28.601 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:28.601 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:28.601 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:28.601 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:28.602 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:19:28.602 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:19:28.602 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:19:28.602 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:28.602 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.621+0000 7f0f99bf4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:28.602 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.622+0000 7f0f927fc640 1 -- 172.21.15.17:0/2437026209 <== mon.1 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f8c001cc0 con 0x7f0f9414a700 2024-03-29T11:19:28.602 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.622+0000 7f0f927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:28.602 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.622+0000 7f0f927fc640 1 -- 172.21.15.17:0/2437026209 <== mon.1 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f0f8c002050 con 0x7f0f9414a700 2024-03-29T11:19:28.602 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.622+0000 7f0f927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:28.602 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.622+0000 7f0f927fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:28.602 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.622+0000 7f0f927fc640 10 monclient: dump: 2024-03-29T11:19:28.602 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:19:28.602 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:19:28.602 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:28.602 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:28.603 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:28.603 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:28.603 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:19:28.603 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:19:28.603 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:19:28.603 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:28.603 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.622+0000 7f0f99bf4640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:28.603 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.622+0000 7f0f99bf4640 4 set_mon_vals no callback set 2024-03-29T11:19:28.605 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.623+0000 7f0f99bf4640 10 monclient: shutdown 2024-03-29T11:19:28.605 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.623+0000 7f0f99bf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:28.605 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.623+0000 7f0f99bf4640 1 -- 172.21.15.17:0/2437026209 >> v1:172.21.15.94:6789/0 conn(0x7f0f9414a700 legacy=0x7f0f940abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:28.605 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.623+0000 7f0f99bf4640 1 -- 172.21.15.17:0/2437026209 shutdown_connections 2024-03-29T11:19:28.605 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.623+0000 7f0f99bf4640 1 -- 172.21.15.17:0/2437026209 >> 172.21.15.17:0/2437026209 conn(0x7f0f9406b660 msgr2=0x7f0f9406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:28.605 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.623+0000 7f0f99bf4640 1 -- 172.21.15.17:0/2437026209 shutdown_connections 2024-03-29T11:19:28.605 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.623+0000 7f0f99bf4640 1 -- 172.21.15.17:0/2437026209 wait complete. 2024-03-29T11:19:28.605 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.624+0000 7f0f99bf4640 10 monclient: build_initial_monmap 2024-03-29T11:19:28.605 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.624+0000 7f0f99bf4640 10 monclient: monmap: 2024-03-29T11:19:28.605 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:19:28.606 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:28.606 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:19:28.606 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:19:28.606 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:28.606 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:28.606 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:19:28.606 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:19:28.606 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:19:28.606 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:28.606 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.625+0000 7f0f99bf4640 1 librados: starting msgr at 2024-03-29T11:19:28.606 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.625+0000 7f0f99bf4640 1 librados: starting objecter 2024-03-29T11:19:28.606 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.625+0000 7f0f99bf4640 1 Processor -- start 2024-03-29T11:19:28.606 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.625+0000 7f0f99bf4640 1 -- start start 2024-03-29T11:19:28.606 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.625+0000 7f0f99bf4640 1 librados: setting wanted keys 2024-03-29T11:19:28.606 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.625+0000 7f0f99bf4640 1 librados: calling monclient init 2024-03-29T11:19:28.607 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.625+0000 7f0f99bf4640 10 monclient: init 2024-03-29T11:19:28.607 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.626+0000 7f0f99bf4640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:28.607 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.626+0000 7f0f99bf4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:28.607 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.626+0000 7f0f99bf4640 10 monclient(hunting): picked mon.noname-a con 0x7f0f940ac5a0 addr v1:172.21.15.17:6789/0 2024-03-29T11:19:28.607 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.626+0000 7f0f99bf4640 10 monclient(hunting): picked mon.noname-b con 0x7f0f940b0770 addr v1:172.21.15.94:6789/0 2024-03-29T11:19:28.607 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.626+0000 7f0f92ffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f0f940b0770 0x7f0f941deea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:46958/0 (socket says 172.21.15.17:46958) 2024-03-29T11:19:28.607 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.626+0000 7f0f92ffd640 1 -- 172.21.15.17:0/3398112455 learned_addr learned my addr 172.21.15.17:0/3398112455 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:19:28.607 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.626+0000 7f0f99bf4640 10 monclient(hunting): picked mon.noname-c con 0x7f0f9414a700 addr v1:172.21.15.17:6790/0 2024-03-29T11:19:28.607 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.626+0000 7f0f99bf4640 1 -- 172.21.15.17:0/3398112455 --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f941239d0 con 0x7f0f940ac5a0 2024-03-29T11:19:28.607 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.627+0000 7f0f99bf4640 1 -- 172.21.15.17:0/3398112455 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f94125cb0 con 0x7f0f940b0770 2024-03-29T11:19:28.607 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.627+0000 7f0f99bf4640 1 -- 172.21.15.17:0/3398112455 --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f94145930 con 0x7f0f9414a700 2024-03-29T11:19:28.607 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.627+0000 7f0f99bf4640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:28.607 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.627+0000 7f0f99bf4640 10 monclient(hunting): authenticate will time out at 874.832110s 2024-03-29T11:19:28.610 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.629+0000 7f0f7bfff640 10 client.?.objecter ms_handle_connect 0x7f0f940b0770 2024-03-29T11:19:28.610 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.629+0000 7f0f7bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:28.610 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.629+0000 7f0f7bfff640 1 -- 172.21.15.17:0/3398112455 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1024914453 0 0) 0x7f0f94125cb0 con 0x7f0f940b0770 2024-03-29T11:19:28.610 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.629+0000 7f0f7bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:28.610 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.629+0000 7f0f7bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:28.610 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.629+0000 7f0f7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:28.610 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.629+0000 7f0f7bfff640 10 monclient(hunting): my global_id is 4261 2024-03-29T11:19:28.610 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.629+0000 7f0f7bfff640 1 -- 172.21.15.17:0/3398112455 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f80004eb0 con 0x7f0f940b0770 2024-03-29T11:19:28.610 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.630+0000 7f0f7bfff640 1 -- 172.21.15.17:0/3398112455 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3957962594 0 0) 0x7f0f80004eb0 con 0x7f0f940b0770 2024-03-29T11:19:28.610 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.630+0000 7f0f7bfff640 1 -- 172.21.15.17:0/3398112455 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f94125cb0 con 0x7f0f940b0770 2024-03-29T11:19:28.610 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.630+0000 7f0f7bfff640 10 client.?.objecter ms_handle_connect 0x7f0f9414a700 2024-03-29T11:19:28.611 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.630+0000 7f0f7bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:28.611 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.630+0000 7f0f7bfff640 1 -- 172.21.15.17:0/3398112455 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f0f8c0019f0 con 0x7f0f940b0770 2024-03-29T11:19:28.611 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.630+0000 7f0f7bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:28.611 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.630+0000 7f0f7bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:28.611 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.630+0000 7f0f7bfff640 10 monclient(hunting): dump: 2024-03-29T11:19:28.611 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:19:28.611 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:19:28.611 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:28.611 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:28.611 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:28.611 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:28.611 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:19:28.611 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:19:28.611 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:19:28.611 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:28.611 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.631+0000 7f0f7bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:28.612 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.631+0000 7f0f7bfff640 1 -- 172.21.15.17:0/3398112455 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1732317837 0 0) 0x7f0f94145930 con 0x7f0f9414a700 2024-03-29T11:19:28.612 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.631+0000 7f0f7bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:28.612 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.631+0000 7f0f7bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:28.612 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.631+0000 7f0f7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:28.612 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.631+0000 7f0f7bfff640 10 monclient(hunting): my global_id is 4256 2024-03-29T11:19:28.612 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.631+0000 7f0f7bfff640 1 -- 172.21.15.17:0/3398112455 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f80004eb0 con 0x7f0f9414a700 2024-03-29T11:19:28.612 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.631+0000 7f0f7bfff640 1 -- 172.21.15.17:0/3398112455 <== mon.1 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1897662940 0 0) 0x7f0f94125cb0 con 0x7f0f940b0770 2024-03-29T11:19:28.612 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.631+0000 7f0f7bfff640 1 -- 172.21.15.17:0/3398112455 >> v1:172.21.15.17:6790/0 conn(0x7f0f9414a700 legacy=0x7f0f941e6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:28.612 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.631+0000 7f0f7bfff640 1 -- 172.21.15.17:0/3398112455 >> v1:172.21.15.17:6789/0 conn(0x7f0f940ac5a0 legacy=0x7f0f94144cb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:19:28.615 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.633+0000 7f0f7bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:19:28.615 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.634+0000 7f0f7bfff640 1 monclient: found mon.b 2024-03-29T11:19:28.615 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.634+0000 7f0f7bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:28.615 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.634+0000 7f0f7bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:19:28.615 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.634+0000 7f0f7bfff640 1 -- 172.21.15.17:0/3398112455 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f941453c0 con 0x7f0f940b0770 2024-03-29T11:19:28.615 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.634+0000 7f0f7bfff640 10 monclient: _finish_auth 0 2024-03-29T11:19:28.615 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.634+0000 7f0f7bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:19:28.615 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.634+0000 7f0f7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:28.615 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.634+0000 7f0f99bf4640 5 monclient: authenticate success, global_id 4261 2024-03-29T11:19:28.615 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.634+0000 7f0f99bf4640 10 monclient: _renew_subs 2024-03-29T11:19:28.615 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.634+0000 7f0f99bf4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:19:28.615 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.634+0000 7f0f99bf4640 1 -- 172.21.15.17:0/3398112455 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f94145610 con 0x7f0f940b0770 2024-03-29T11:19:28.615 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.634+0000 7f0f7bfff640 1 -- 172.21.15.17:0/3398112455 <== mon.1 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f8c004e80 con 0x7f0f940b0770 2024-03-29T11:19:28.615 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.634+0000 7f0f99bf4640 10 client.4261.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:28.616 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.634+0000 7f0f99bf4640 10 monclient: _renew_subs 2024-03-29T11:19:28.616 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.634+0000 7f0f99bf4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:19:28.616 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.634+0000 7f0f99bf4640 1 -- 172.21.15.17:0/3398112455 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f94126b00 con 0x7f0f940b0770 2024-03-29T11:19:28.616 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.634+0000 7f0f99bf4640 1 librados: init done 2024-03-29T11:19:28.616 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.634+0000 7f0f7bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:28.616 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.634+0000 7f0f7bfff640 1 -- 172.21.15.17:0/3398112455 <== mon.1 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f0f8c005250 con 0x7f0f940b0770 2024-03-29T11:19:28.616 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.634+0000 7f0f7bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:28.616 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.634+0000 7f0f7bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:28.616 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.634+0000 7f0f7bfff640 10 monclient: dump: 2024-03-29T11:19:28.616 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:19:28.616 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:19:28.616 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:28.616 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:28.616 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:28.616 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:28.617 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:19:28.617 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:19:28.617 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:19:28.617 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:28.617 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.635+0000 7f0f917fa640 4 set_mon_vals no callback set 2024-03-29T11:19:28.617 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.636+0000 7f0f99bf4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:28.617 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.636+0000 7f0f99bf4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:28.617 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.636+0000 7f0f99bf4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:19:28.617 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.636+0000 7f0f99bf4640 1 -- 172.21.15.17:0/3398112455 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f74005080 con 0x7f0f940b0770 2024-03-29T11:19:28.618 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.638+0000 7f0f7bfff640 1 -- 172.21.15.17:0/3398112455 <== mon.1 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f0f8c0053b0 con 0x7f0f940b0770 2024-03-29T11:19:28.618 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.639+0000 7f0f7bfff640 1 -- 172.21.15.17:0/3398112455 <== mon.1 v1:172.21.15.94:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f0f8c084f00 con 0x7f0f940b0770 2024-03-29T11:19:28.618 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.639+0000 7f0f7bfff640 10 client.4261.objecter ms_dispatch 0x7f0f94000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:28.618 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.639+0000 7f0f7bfff640 3 client.4261.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:28.618 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.639+0000 7f0f7bfff640 3 client.4261.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:28.618 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.639+0000 7f0f7bfff640 20 client.4261.objecter dump_active .. 0 homeless 2024-03-29T11:19:28.618 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.639+0000 7f0f937fe640 10 monclient: get_auth_request con 0x7f0f8006dd80 auth_method 0 2024-03-29T11:19:28.618 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.640+0000 7f0f7bfff640 10 client.4261.objecter ms_handle_connect 0x7f0f8006dd80 2024-03-29T11:19:28.623 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.646+0000 7f0f7bfff640 1 -- 172.21.15.17:0/3398112455 <== mon.1 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0f8c0507e0 con 0x7f0f940b0770 2024-03-29T11:19:28.623 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.646+0000 7f0f7bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:28.623 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.646+0000 7f0f7bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:28.956 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.979+0000 7f0f99bf4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:28.956 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.979+0000 7f0f99bf4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:28.956 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.979+0000 7f0f99bf4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:19:28.956 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.979+0000 7f0f99bf4640 1 -- 172.21.15.17:0/3398112455 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f74005330 con 0x7f0f940b0770 2024-03-29T11:19:28.957 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.980+0000 7f0f7bfff640 1 -- 172.21.15.17:0/3398112455 <== mon.1 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f0f8c052710 con 0x7f0f940b0770 2024-03-29T11:19:28.957 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.980+0000 7f0f7bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:28.957 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.980+0000 7f0f7bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:28.957 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:19:28.957 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:28.960 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.983+0000 7f0f99bf4640 10 librados: watch_flush enter 2024-03-29T11:19:28.961 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.983+0000 7f0f99bf4640 10 librados: watch_flush exit 2024-03-29T11:19:28.961 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.983+0000 7f0f99bf4640 20 client.4261.objecter shutdown clearing up homeless session... 2024-03-29T11:19:28.961 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.983+0000 7f0f99bf4640 10 client.4261.objecter successfully canceled tick 2024-03-29T11:19:28.961 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.983+0000 7f0f99bf4640 1 -- 172.21.15.17:0/3398112455 >> v1:172.21.15.17:6816/3236185052 conn(0x7f0f8006dd80 legacy=0x7f0f80070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:28.961 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.983+0000 7f0f99bf4640 10 monclient: shutdown 2024-03-29T11:19:28.961 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.983+0000 7f0f99bf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:28.961 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.983+0000 7f0f99bf4640 1 -- 172.21.15.17:0/3398112455 >> v1:172.21.15.94:6789/0 conn(0x7f0f940b0770 legacy=0x7f0f941deea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:28.961 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.983+0000 7f0f99bf4640 1 -- 172.21.15.17:0/3398112455 shutdown_connections 2024-03-29T11:19:28.961 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.983+0000 7f0f99bf4640 1 -- 172.21.15.17:0/3398112455 >> 172.21.15.17:0/3398112455 conn(0x7f0f9406b660 msgr2=0x7f0f9406ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:28.961 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.983+0000 7f0f99bf4640 1 -- 172.21.15.17:0/3398112455 shutdown_connections 2024-03-29T11:19:28.961 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.983+0000 7f0f99bf4640 1 -- 172.21.15.17:0/3398112455 wait complete. 2024-03-29T11:19:28.961 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:28.983+0000 7f0f99bf4640 1 librados: shutdown 2024-03-29T11:19:28.977 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:35.783 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:35.930 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.952+0000 7f86dece6640 10 monclient: get_monmap_and_config 2024-03-29T11:19:35.930 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.952+0000 7f86dece6640 10 monclient: build_initial_monmap 2024-03-29T11:19:35.930 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.952+0000 7f86dece6640 10 monclient: monmap: 2024-03-29T11:19:35.930 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:19:35.930 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:19:35.930 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:19:35.953561+0000 2024-03-29T11:19:35.931 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:19:35.953561+0000 2024-03-29T11:19:35.931 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:35.931 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:35.931 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:19:35.931 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:19:35.931 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:19:35.931 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:35.931 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.955+0000 7f86dece6640 1 Processor -- start 2024-03-29T11:19:35.933 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.955+0000 7f86dece6640 1 -- start start 2024-03-29T11:19:35.933 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.955+0000 7f86dece6640 10 monclient: init 2024-03-29T11:19:35.933 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.956+0000 7f86dece6640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:35.933 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.956+0000 7f86dece6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:35.933 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.956+0000 7f86dece6640 10 monclient(hunting): picked mon.noname-c con 0x7f86d80a9ac0 addr v1:172.21.15.94:6789/0 2024-03-29T11:19:35.933 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.956+0000 7f86dece6640 10 monclient(hunting): picked mon.noname-b con 0x7f86d813ee90 addr v1:172.21.15.17:6790/0 2024-03-29T11:19:35.934 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.956+0000 7f86dece6640 10 monclient(hunting): picked mon.noname-a con 0x7f86d80af900 addr v1:172.21.15.17:6789/0 2024-03-29T11:19:35.934 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.956+0000 7f86dece6640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86d80afd10 con 0x7f86d80af900 2024-03-29T11:19:35.934 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.956+0000 7f86dece6640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86d80af6f0 con 0x7f86d80a9ac0 2024-03-29T11:19:35.934 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.956+0000 7f86dece6640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86d80aac80 con 0x7f86d813ee90 2024-03-29T11:19:35.934 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.956+0000 7f86dece6640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:35.934 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.956+0000 7f86dece6640 10 monclient(hunting): authenticate will time out at 882.161211s 2024-03-29T11:19:35.936 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.959+0000 7f86dd283640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f86d80af900 0x7f86d8149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:56897/0 (socket says 172.21.15.17:56897) 2024-03-29T11:19:35.936 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.959+0000 7f86dd283640 1 -- 172.21.15.17:0/3680027413 learned_addr learned my addr 172.21.15.17:0/3680027413 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:19:35.937 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.960+0000 7f86d77fe640 1 -- 172.21.15.17:0/3680027413 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1795157213 0 0) 0x7f86d80afd10 con 0x7f86d80af900 2024-03-29T11:19:35.937 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.960+0000 7f86d77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:35.937 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.960+0000 7f86d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:35.937 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.960+0000 7f86d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:35.937 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.960+0000 7f86d77fe640 10 monclient(hunting): my global_id is 4242 2024-03-29T11:19:35.937 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.960+0000 7f86d77fe640 1 -- 172.21.15.17:0/3680027413 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86c0003520 con 0x7f86d80af900 2024-03-29T11:19:35.937 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.960+0000 7f86d77fe640 1 -- 172.21.15.17:0/3680027413 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1366610954 0 0) 0x7f86c0003520 con 0x7f86d80af900 2024-03-29T11:19:35.937 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.960+0000 7f86d77fe640 1 -- 172.21.15.17:0/3680027413 >> v1:172.21.15.17:6790/0 conn(0x7f86d813ee90 legacy=0x7f86d813b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:19:35.939 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.962+0000 7f86d77fe640 1 -- 172.21.15.17:0/3680027413 >> v1:172.21.15.94:6789/0 conn(0x7f86d80a9ac0 legacy=0x7f86d80adfe0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:19:35.939 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.962+0000 7f86d77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:35.939 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.962+0000 7f86d77fe640 1 monclient: found mon.noname-a 2024-03-29T11:19:35.939 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.962+0000 7f86d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:35.939 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.962+0000 7f86d77fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.17:6789/0 2024-03-29T11:19:35.939 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.962+0000 7f86d77fe640 1 -- 172.21.15.17:0/3680027413 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86d80aaf60 con 0x7f86d80af900 2024-03-29T11:19:35.939 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.962+0000 7f86d77fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:35.939 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.962+0000 7f86d77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:35.940 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.962+0000 7f86d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:35.940 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.962+0000 7f86d77fe640 1 -- 172.21.15.17:0/3680027413 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f86cc001690 con 0x7f86d80af900 2024-03-29T11:19:35.940 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.962+0000 7f86d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:35.940 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.962+0000 7f86d77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:35.940 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.962+0000 7f86d77fe640 10 monclient: dump: 2024-03-29T11:19:35.940 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:19:35.940 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:19:35.940 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:35.940 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:35.940 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:35.940 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:35.940 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:19:35.940 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:19:35.940 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:19:35.940 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:35.942 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.963+0000 7f86d77fe640 1 -- 172.21.15.17:0/3680027413 <== mon.0 v1:172.21.15.17:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86cc001860 con 0x7f86d80af900 2024-03-29T11:19:35.942 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.963+0000 7f86d77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:35.942 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.963+0000 7f86d77fe640 1 -- 172.21.15.17:0/3680027413 <== mon.0 v1:172.21.15.17:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f86cc001cb0 con 0x7f86d80af900 2024-03-29T11:19:35.942 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.963+0000 7f86d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:35.942 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.963+0000 7f86d77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:35.942 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.963+0000 7f86d77fe640 10 monclient: dump: 2024-03-29T11:19:35.942 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:19:35.942 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:19:35.942 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:35.942 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:35.942 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:35.942 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:35.942 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:19:35.942 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:19:35.942 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:19:35.943 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:35.943 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.963+0000 7f86dece6640 5 monclient: authenticate success, global_id 4242 2024-03-29T11:19:35.943 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.963+0000 7f86dece6640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:35.943 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.963+0000 7f86dece6640 4 set_mon_vals no callback set 2024-03-29T11:19:35.943 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.964+0000 7f86dece6640 10 monclient: shutdown 2024-03-29T11:19:35.943 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.964+0000 7f86dece6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:35.943 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.964+0000 7f86dece6640 1 -- 172.21.15.17:0/3680027413 >> v1:172.21.15.17:6789/0 conn(0x7f86d80af900 legacy=0x7f86d8149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:35.943 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.964+0000 7f86dece6640 1 -- 172.21.15.17:0/3680027413 shutdown_connections 2024-03-29T11:19:35.943 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.964+0000 7f86dece6640 1 -- 172.21.15.17:0/3680027413 >> 172.21.15.17:0/3680027413 conn(0x7f86d80b35e0 msgr2=0x7f86d80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:35.943 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.964+0000 7f86dece6640 1 -- 172.21.15.17:0/3680027413 shutdown_connections 2024-03-29T11:19:35.943 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.964+0000 7f86dece6640 1 -- 172.21.15.17:0/3680027413 wait complete. 2024-03-29T11:19:35.944 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.965+0000 7f86dece6640 10 monclient: build_initial_monmap 2024-03-29T11:19:35.944 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.965+0000 7f86dece6640 10 monclient: monmap: 2024-03-29T11:19:35.944 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:19:35.944 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:35.944 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:19:35.944 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:19:35.944 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:35.944 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:35.944 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:19:35.944 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:19:35.944 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:19:35.944 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:35.945 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.966+0000 7f86dece6640 1 librados: starting msgr at 2024-03-29T11:19:35.945 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.966+0000 7f86dece6640 1 librados: starting objecter 2024-03-29T11:19:35.945 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.966+0000 7f86dece6640 1 Processor -- start 2024-03-29T11:19:35.945 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.966+0000 7f86dece6640 1 -- start start 2024-03-29T11:19:35.945 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.966+0000 7f86dece6640 1 librados: setting wanted keys 2024-03-29T11:19:35.945 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.966+0000 7f86dece6640 1 librados: calling monclient init 2024-03-29T11:19:35.945 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.966+0000 7f86dece6640 10 monclient: init 2024-03-29T11:19:35.945 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.967+0000 7f86dece6640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:35.945 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.967+0000 7f86dece6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:35.945 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.967+0000 7f86dece6640 10 monclient(hunting): picked mon.noname-a con 0x7f86d80a9ac0 addr v1:172.21.15.17:6789/0 2024-03-29T11:19:35.945 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.967+0000 7f86dece6640 10 monclient(hunting): picked mon.noname-c con 0x7f86d80af900 addr v1:172.21.15.17:6790/0 2024-03-29T11:19:35.945 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.967+0000 7f86dece6640 10 monclient(hunting): picked mon.noname-b con 0x7f86d813ee90 addr v1:172.21.15.94:6789/0 2024-03-29T11:19:35.945 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.967+0000 7f86dece6640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86d81dc590 con 0x7f86d80a9ac0 2024-03-29T11:19:35.945 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.967+0000 7f86dece6640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86d81dc240 con 0x7f86d813ee90 2024-03-29T11:19:35.946 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.967+0000 7f86dece6640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86d8072490 con 0x7f86d80af900 2024-03-29T11:19:35.946 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.967+0000 7f86dece6640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:35.946 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.967+0000 7f86dece6640 10 monclient(hunting): authenticate will time out at 882.172208s 2024-03-29T11:19:35.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.970+0000 7f86d7fff640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f86d80af900 0x7f86d8137e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:45955/0 (socket says 172.21.15.17:45955) 2024-03-29T11:19:35.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.971+0000 7f86d7fff640 1 -- 172.21.15.17:0/3811270370 learned_addr learned my addr 172.21.15.17:0/3811270370 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:19:35.948 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.972+0000 7f86d57fa640 10 client.?.objecter ms_handle_connect 0x7f86d80af900 2024-03-29T11:19:35.949 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.972+0000 7f86d57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:35.949 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.973+0000 7f86d57fa640 1 -- 172.21.15.17:0/3811270370 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3194615431 0 0) 0x7f86d8072490 con 0x7f86d80af900 2024-03-29T11:19:35.949 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.973+0000 7f86d57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:19:35.949 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.973+0000 7f86d57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:35.949 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.973+0000 7f86d57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:35.950 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.973+0000 7f86d57fa640 10 monclient(hunting): my global_id is 4259 2024-03-29T11:19:35.950 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.973+0000 7f86d57fa640 1 -- 172.21.15.17:0/3811270370 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86c0003520 con 0x7f86d80af900 2024-03-29T11:19:35.952 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.973+0000 7f86d57fa640 1 -- 172.21.15.17:0/3811270370 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2758975786 0 0) 0x7f86c0003520 con 0x7f86d80af900 2024-03-29T11:19:35.952 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.973+0000 7f86d57fa640 1 -- 172.21.15.17:0/3811270370 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86d8072490 con 0x7f86d80af900 2024-03-29T11:19:35.952 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.973+0000 7f86d57fa640 1 -- 172.21.15.17:0/3811270370 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f86c8001960 con 0x7f86d80af900 2024-03-29T11:19:35.952 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.973+0000 7f86d57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:35.953 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.973+0000 7f86d57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:35.953 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.973+0000 7f86d57fa640 10 monclient(hunting): dump: 2024-03-29T11:19:35.953 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:19:35.953 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:19:35.953 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:35.953 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:35.953 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:35.953 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:35.953 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:19:35.953 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:19:35.953 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:19:35.953 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:35.953 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.973+0000 7f86d57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:35.953 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.973+0000 7f86d57fa640 1 -- 172.21.15.17:0/3811270370 <== mon.2 v1:172.21.15.17:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1895667398 0 0) 0x7f86d8072490 con 0x7f86d80af900 2024-03-29T11:19:35.953 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.973+0000 7f86d57fa640 1 -- 172.21.15.17:0/3811270370 >> v1:172.21.15.94:6789/0 conn(0x7f86d813ee90 legacy=0x7f86d806ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:35.954 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.973+0000 7f86d57fa640 1 -- 172.21.15.17:0/3811270370 >> v1:172.21.15.17:6789/0 conn(0x7f86d80a9ac0 legacy=0x7f86d81dbb30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:19:35.954 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.974+0000 7f86d57fa640 10 monclient: _finish_hunting 0 2024-03-29T11:19:35.954 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.974+0000 7f86d57fa640 1 monclient: found mon.c 2024-03-29T11:19:35.954 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.974+0000 7f86d57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:35.954 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.974+0000 7f86d57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:19:35.954 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.974+0000 7f86d57fa640 1 -- 172.21.15.17:0/3811270370 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86d8141790 con 0x7f86d80af900 2024-03-29T11:19:35.954 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.974+0000 7f86d57fa640 10 monclient: _finish_auth 0 2024-03-29T11:19:35.954 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.974+0000 7f86d57fa640 10 monclient: _check_auth_tickets 2024-03-29T11:19:35.954 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.974+0000 7f86d57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:35.954 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.974+0000 7f86dece6640 5 monclient: authenticate success, global_id 4259 2024-03-29T11:19:35.954 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.974+0000 7f86dece6640 10 monclient: _renew_subs 2024-03-29T11:19:35.954 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.974+0000 7f86dece6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:19:35.954 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.974+0000 7f86dece6640 1 -- 172.21.15.17:0/3811270370 --> v1:172.21.15.17:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86d8141a40 con 0x7f86d80af900 2024-03-29T11:19:35.954 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.974+0000 7f86dece6640 10 client.4259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:35.954 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.974+0000 7f86dece6640 10 monclient: _renew_subs 2024-03-29T11:19:35.955 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.974+0000 7f86dece6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:19:35.955 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.974+0000 7f86dece6640 1 -- 172.21.15.17:0/3811270370 --> v1:172.21.15.17:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f86d8138770 con 0x7f86d80af900 2024-03-29T11:19:35.955 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.974+0000 7f86dece6640 1 librados: init done 2024-03-29T11:19:35.955 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.975+0000 7f86d57fa640 1 -- 172.21.15.17:0/3811270370 <== mon.2 v1:172.21.15.17:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86c8001c80 con 0x7f86d80af900 2024-03-29T11:19:35.955 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.975+0000 7f86d57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:35.955 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.975+0000 7f86d57fa640 1 -- 172.21.15.17:0/3811270370 <== mon.2 v1:172.21.15.17:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f86c8003a40 con 0x7f86d80af900 2024-03-29T11:19:35.955 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.975+0000 7f86d6ffd640 4 set_mon_vals no callback set 2024-03-29T11:19:35.955 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.976+0000 7f86d57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:35.955 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.976+0000 7f86d57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:35.955 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.976+0000 7f86d57fa640 10 monclient: dump: 2024-03-29T11:19:35.955 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:19:35.955 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:19:35.955 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:35.955 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:35.955 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:35.955 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:35.955 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:19:35.956 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:19:35.956 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:19:35.956 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:35.957 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.976+0000 7f86dece6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:35.957 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.976+0000 7f86dece6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:35.957 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.976+0000 7f86dece6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:19:35.957 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.976+0000 7f86dece6640 1 -- 172.21.15.17:0/3811270370 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86b8005080 con 0x7f86d80af900 2024-03-29T11:19:35.957 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.978+0000 7f86d57fa640 1 -- 172.21.15.17:0/3811270370 <== mon.2 v1:172.21.15.17:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f86c8003ba0 con 0x7f86d80af900 2024-03-29T11:19:35.957 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.978+0000 7f86d57fa640 1 -- 172.21.15.17:0/3811270370 <== mon.2 v1:172.21.15.17:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f86c8084b40 con 0x7f86d80af900 2024-03-29T11:19:35.957 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.979+0000 7f86d57fa640 10 client.4259.objecter ms_dispatch 0x7f86d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:35.957 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.979+0000 7f86d57fa640 3 client.4259.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:35.957 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.979+0000 7f86d57fa640 3 client.4259.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:35.958 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.979+0000 7f86d57fa640 20 client.4259.objecter dump_active .. 0 homeless 2024-03-29T11:19:35.958 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.979+0000 7f86dca82640 10 monclient: get_auth_request con 0x7f86c006dea0 auth_method 0 2024-03-29T11:19:35.958 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.979+0000 7f86d57fa640 10 client.4259.objecter ms_handle_connect 0x7f86c006dea0 2024-03-29T11:19:35.966 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.988+0000 7f86d57fa640 1 -- 172.21.15.17:0/3811270370 <== mon.2 v1:172.21.15.17:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f86c8050420 con 0x7f86d80af900 2024-03-29T11:19:35.966 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.989+0000 7f86d57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:35.966 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:35.989+0000 7f86d57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:36.297 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:36.320+0000 7f86dece6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:36.297 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:36.320+0000 7f86dece6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:36.297 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:36.320+0000 7f86dece6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:19:36.297 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:36.320+0000 7f86dece6640 1 -- 172.21.15.17:0/3811270370 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f86b8005330 con 0x7f86d80af900 2024-03-29T11:19:36.300 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:36.323+0000 7f86d57fa640 1 -- 172.21.15.17:0/3811270370 <== mon.2 v1:172.21.15.17:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f86c8052350 con 0x7f86d80af900 2024-03-29T11:19:36.300 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:36.323+0000 7f86d57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:36.300 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:36.323+0000 7f86d57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:36.300 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:19:36.300 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:36.304 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:36.327+0000 7f86dece6640 10 librados: watch_flush enter 2024-03-29T11:19:36.304 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:36.327+0000 7f86dece6640 10 librados: watch_flush exit 2024-03-29T11:19:36.304 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:36.327+0000 7f86dece6640 20 client.4259.objecter shutdown clearing up homeless session... 2024-03-29T11:19:36.304 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:36.327+0000 7f86dece6640 10 client.4259.objecter successfully canceled tick 2024-03-29T11:19:36.304 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:36.327+0000 7f86dece6640 1 -- 172.21.15.17:0/3811270370 >> v1:172.21.15.17:6816/3236185052 conn(0x7f86c006dea0 legacy=0x7f86c0070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:36.304 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:36.327+0000 7f86dece6640 10 monclient: shutdown 2024-03-29T11:19:36.304 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:36.327+0000 7f86dece6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:36.304 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:36.327+0000 7f86dece6640 1 -- 172.21.15.17:0/3811270370 >> v1:172.21.15.17:6790/0 conn(0x7f86d80af900 legacy=0x7f86d8137e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:36.305 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:36.328+0000 7f86dece6640 1 -- 172.21.15.17:0/3811270370 shutdown_connections 2024-03-29T11:19:36.305 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:36.328+0000 7f86dece6640 1 -- 172.21.15.17:0/3811270370 >> 172.21.15.17:0/3811270370 conn(0x7f86d80b35e0 msgr2=0x7f86d813b100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:36.305 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:36.328+0000 7f86dece6640 1 -- 172.21.15.17:0/3811270370 shutdown_connections 2024-03-29T11:19:36.305 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:36.328+0000 7f86dece6640 1 -- 172.21.15.17:0/3811270370 wait complete. 2024-03-29T11:19:36.305 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:36.328+0000 7f86dece6640 1 librados: shutdown 2024-03-29T11:19:36.322 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:43.129 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:43.276 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.299+0000 7fa9c759e640 10 monclient: get_monmap_and_config 2024-03-29T11:19:43.276 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.299+0000 7fa9c759e640 10 monclient: build_initial_monmap 2024-03-29T11:19:43.276 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.299+0000 7fa9c759e640 10 monclient: monmap: 2024-03-29T11:19:43.276 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:19:43.276 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:19:43.276 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:19:43.300523+0000 2024-03-29T11:19:43.276 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:19:43.300523+0000 2024-03-29T11:19:43.277 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:43.277 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:43.277 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:19:43.277 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:19:43.277 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:19:43.277 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:43.283 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.303+0000 7fa9c759e640 1 Processor -- start 2024-03-29T11:19:43.283 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.303+0000 7fa9c759e640 1 -- start start 2024-03-29T11:19:43.283 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.303+0000 7fa9c759e640 10 monclient: init 2024-03-29T11:19:43.283 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.304+0000 7fa9c759e640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:43.283 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.304+0000 7fa9c759e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:43.284 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.304+0000 7fa9c759e640 10 monclient(hunting): picked mon.noname-c con 0x7fa9c0006e30 addr v1:172.21.15.94:6789/0 2024-03-29T11:19:43.284 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.304+0000 7fa9c759e640 10 monclient(hunting): picked mon.noname-a con 0x7fa9c0035e00 addr v1:172.21.15.17:6789/0 2024-03-29T11:19:43.285 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.308+0000 7fa9c759e640 10 monclient(hunting): picked mon.noname-b con 0x7fa9c00c9670 addr v1:172.21.15.17:6790/0 2024-03-29T11:19:43.285 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.308+0000 7fa9c759e640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c0035970 con 0x7fa9c0035e00 2024-03-29T11:19:43.285 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.308+0000 7fa9c759e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c0036280 con 0x7fa9c0006e30 2024-03-29T11:19:43.285 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.308+0000 7fa9c759e640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c0035640 con 0x7fa9c00c9670 2024-03-29T11:19:43.285 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.308+0000 7fa9c759e640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:43.285 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.308+0000 7fa9c759e640 10 monclient(hunting): authenticate will time out at 889.513487s 2024-03-29T11:19:43.285 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.308+0000 7fa9c659c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fa9c0006e30 0x7fa9c00cbcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:46073/0 (socket says 172.21.15.17:46073) 2024-03-29T11:19:43.285 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.308+0000 7fa9c659c640 1 -- 172.21.15.17:0/3667773062 learned_addr learned my addr 172.21.15.17:0/3667773062 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:19:43.288 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.310+0000 7fa9c559a640 1 -- 172.21.15.17:0/3667773062 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 653315675 0 0) 0x7fa9c0036280 con 0x7fa9c0006e30 2024-03-29T11:19:43.288 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.310+0000 7fa9c559a640 10 monclient(hunting): _negotiate 2024-03-29T11:19:43.288 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.310+0000 7fa9c559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:43.288 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.310+0000 7fa9c559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:43.288 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.310+0000 7fa9c559a640 10 monclient(hunting): my global_id is 4264 2024-03-29T11:19:43.288 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.310+0000 7fa9c559a640 1 -- 172.21.15.17:0/3667773062 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9b0003520 con 0x7fa9c0006e30 2024-03-29T11:19:43.288 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.311+0000 7fa9c559a640 1 -- 172.21.15.17:0/3667773062 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 661461591 0 0) 0x7fa9b0003520 con 0x7fa9c0006e30 2024-03-29T11:19:43.288 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.311+0000 7fa9c559a640 1 -- 172.21.15.17:0/3667773062 >> v1:172.21.15.17:6790/0 conn(0x7fa9c00c9670 legacy=0x7fa9c00d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:19:43.288 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.311+0000 7fa9c559a640 1 -- 172.21.15.17:0/3667773062 >> v1:172.21.15.17:6789/0 conn(0x7fa9c0035e00 legacy=0x7fa9c00c8f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:19:43.292 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.312+0000 7fa9c559a640 10 monclient: _finish_hunting 0 2024-03-29T11:19:43.292 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.312+0000 7fa9c559a640 1 monclient: found mon.noname-c 2024-03-29T11:19:43.292 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.312+0000 7fa9c559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:43.292 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.312+0000 7fa9c559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6789/0 2024-03-29T11:19:43.292 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.312+0000 7fa9c559a640 1 -- 172.21.15.17:0/3667773062 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c0034e20 con 0x7fa9c0006e30 2024-03-29T11:19:43.292 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.312+0000 7fa9c559a640 10 monclient: _finish_auth 0 2024-03-29T11:19:43.292 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.312+0000 7fa9c559a640 10 monclient: _check_auth_tickets 2024-03-29T11:19:43.292 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.312+0000 7fa9c559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:43.292 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.312+0000 7fa9c559a640 1 -- 172.21.15.17:0/3667773062 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fa9bc001690 con 0x7fa9c0006e30 2024-03-29T11:19:43.292 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.312+0000 7fa9c759e640 5 monclient: authenticate success, global_id 4264 2024-03-29T11:19:43.292 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.312+0000 7fa9c759e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:43.292 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.312+0000 7fa9c559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:43.292 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.312+0000 7fa9c559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:43.292 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.312+0000 7fa9c559a640 10 monclient: dump: 2024-03-29T11:19:43.293 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:19:43.293 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:19:43.293 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:43.293 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:43.293 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:43.293 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:43.293 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:19:43.293 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:19:43.293 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:19:43.293 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:43.293 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.312+0000 7fa9c759e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:43.293 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.312+0000 7fa9c559a640 1 -- 172.21.15.17:0/3667773062 <== mon.1 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9bc001860 con 0x7fa9c0006e30 2024-03-29T11:19:43.293 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.312+0000 7fa9c559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:43.293 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.312+0000 7fa9c759e640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:43.293 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.312+0000 7fa9c759e640 4 set_mon_vals no callback set 2024-03-29T11:19:43.294 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.313+0000 7fa9c559a640 1 -- 172.21.15.17:0/3667773062 <== mon.1 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fa9bc001cb0 con 0x7fa9c0006e30 2024-03-29T11:19:43.294 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.313+0000 7fa9c559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:43.294 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.313+0000 7fa9c559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:43.294 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.313+0000 7fa9c559a640 10 monclient: dump: 2024-03-29T11:19:43.294 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:19:43.294 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:19:43.294 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:43.294 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:43.294 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:43.294 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:43.294 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:19:43.294 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:19:43.294 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:19:43.294 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:43.295 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.313+0000 7fa9c759e640 10 monclient: shutdown 2024-03-29T11:19:43.295 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.313+0000 7fa9c759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:43.295 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.313+0000 7fa9c759e640 1 -- 172.21.15.17:0/3667773062 >> v1:172.21.15.94:6789/0 conn(0x7fa9c0006e30 legacy=0x7fa9c00cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:43.295 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.313+0000 7fa9c759e640 1 -- 172.21.15.17:0/3667773062 shutdown_connections 2024-03-29T11:19:43.295 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.313+0000 7fa9c759e640 1 -- 172.21.15.17:0/3667773062 >> 172.21.15.17:0/3667773062 conn(0x7fa9c0005c00 msgr2=0x7fa9c0006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:43.295 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.313+0000 7fa9c759e640 1 -- 172.21.15.17:0/3667773062 shutdown_connections 2024-03-29T11:19:43.295 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.313+0000 7fa9c759e640 1 -- 172.21.15.17:0/3667773062 wait complete. 2024-03-29T11:19:43.295 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.314+0000 7fa9c759e640 10 monclient: build_initial_monmap 2024-03-29T11:19:43.295 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.314+0000 7fa9c759e640 10 monclient: monmap: 2024-03-29T11:19:43.295 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:19:43.295 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:43.295 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:19:43.295 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:19:43.295 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:43.296 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:43.296 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:19:43.296 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:19:43.296 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:19:43.296 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:43.296 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.315+0000 7fa9c759e640 1 librados: starting msgr at 2024-03-29T11:19:43.296 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.315+0000 7fa9c759e640 1 librados: starting objecter 2024-03-29T11:19:43.296 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.315+0000 7fa9c759e640 1 Processor -- start 2024-03-29T11:19:43.297 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.318+0000 7fa9c759e640 1 -- start start 2024-03-29T11:19:43.297 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.318+0000 7fa9c759e640 1 librados: setting wanted keys 2024-03-29T11:19:43.297 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.318+0000 7fa9c759e640 1 librados: calling monclient init 2024-03-29T11:19:43.297 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.318+0000 7fa9c759e640 10 monclient: init 2024-03-29T11:19:43.297 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.319+0000 7fa9c759e640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:43.297 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.319+0000 7fa9c759e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:43.297 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.319+0000 7fa9c759e640 10 monclient(hunting): picked mon.noname-c con 0x7fa9c0006e30 addr v1:172.21.15.17:6790/0 2024-03-29T11:19:43.297 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.319+0000 7fa9c759e640 10 monclient(hunting): picked mon.noname-b con 0x7fa9c0035e00 addr v1:172.21.15.94:6789/0 2024-03-29T11:19:43.297 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.319+0000 7fa9c759e640 10 monclient(hunting): picked mon.noname-a con 0x7fa9c00c9670 addr v1:172.21.15.17:6789/0 2024-03-29T11:19:43.297 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.319+0000 7fa9c759e640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c0164880 con 0x7fa9c00c9670 2024-03-29T11:19:43.297 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.319+0000 7fa9c759e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c01613e0 con 0x7fa9c0035e00 2024-03-29T11:19:43.297 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.319+0000 7fa9c759e640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c00c4a40 con 0x7fa9c0006e30 2024-03-29T11:19:43.297 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.319+0000 7fa9c759e640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:43.297 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.319+0000 7fa9c759e640 10 monclient(hunting): authenticate will time out at 889.524154s 2024-03-29T11:19:43.302 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.324+0000 7fa9c659c640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7fa9c0006e30 0x7fa9c0160cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:56117/0 (socket says 172.21.15.17:56117) 2024-03-29T11:19:43.302 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.324+0000 7fa9c659c640 1 -- 172.21.15.17:0/394084113 learned_addr learned my addr 172.21.15.17:0/394084113 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:19:43.302 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.324+0000 7fa9b6ffd640 10 client.?.objecter ms_handle_connect 0x7fa9c0006e30 2024-03-29T11:19:43.302 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.324+0000 7fa9b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:43.302 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.324+0000 7fa9b6ffd640 1 -- 172.21.15.17:0/394084113 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3941543400 0 0) 0x7fa9c00c4a40 con 0x7fa9c0006e30 2024-03-29T11:19:43.302 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.324+0000 7fa9b6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:43.302 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.325+0000 7fa9b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:43.302 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.325+0000 7fa9b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:43.302 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.325+0000 7fa9b6ffd640 10 monclient(hunting): my global_id is 4262 2024-03-29T11:19:43.303 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.325+0000 7fa9b6ffd640 1 -- 172.21.15.17:0/394084113 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9a8004eb0 con 0x7fa9c0006e30 2024-03-29T11:19:43.303 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.325+0000 7fa9b6ffd640 1 -- 172.21.15.17:0/394084113 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2928891513 0 0) 0x7fa9a8004eb0 con 0x7fa9c0006e30 2024-03-29T11:19:43.303 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.325+0000 7fa9b6ffd640 1 -- 172.21.15.17:0/394084113 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9c00c4a40 con 0x7fa9c0006e30 2024-03-29T11:19:43.306 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.326+0000 7fa9b6ffd640 10 client.?.objecter ms_handle_connect 0x7fa9c00c9670 2024-03-29T11:19:43.306 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.326+0000 7fa9b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:43.306 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.326+0000 7fa9b6ffd640 10 client.?.objecter ms_handle_connect 0x7fa9c0035e00 2024-03-29T11:19:43.306 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.326+0000 7fa9b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:43.306 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.326+0000 7fa9b6ffd640 1 -- 172.21.15.17:0/394084113 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fa9bc001c30 con 0x7fa9c0006e30 2024-03-29T11:19:43.306 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.326+0000 7fa9b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:43.306 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.327+0000 7fa9b6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:43.307 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.327+0000 7fa9b6ffd640 10 monclient(hunting): dump: 2024-03-29T11:19:43.307 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:19:43.307 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:19:43.307 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:43.307 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:43.307 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:43.307 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:43.307 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:19:43.307 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:19:43.307 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:19:43.307 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:43.307 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.327+0000 7fa9b6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:43.307 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.327+0000 7fa9b6ffd640 1 -- 172.21.15.17:0/394084113 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2042510604 0 0) 0x7fa9c0164880 con 0x7fa9c00c9670 2024-03-29T11:19:43.307 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.327+0000 7fa9b6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:43.307 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.327+0000 7fa9b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:43.308 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.327+0000 7fa9b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:43.308 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.327+0000 7fa9b6ffd640 10 monclient(hunting): my global_id is 4245 2024-03-29T11:19:43.308 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.327+0000 7fa9b6ffd640 1 -- 172.21.15.17:0/394084113 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9a8004eb0 con 0x7fa9c00c9670 2024-03-29T11:19:43.308 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.327+0000 7fa9b6ffd640 1 -- 172.21.15.17:0/394084113 <== mon.2 v1:172.21.15.17:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2295538901 0 0) 0x7fa9c00c4a40 con 0x7fa9c0006e30 2024-03-29T11:19:43.308 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.327+0000 7fa9b6ffd640 1 -- 172.21.15.17:0/394084113 >> v1:172.21.15.94:6789/0 conn(0x7fa9c0035e00 legacy=0x7fa9c00cf420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:43.308 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.327+0000 7fa9b6ffd640 1 -- 172.21.15.17:0/394084113 >> v1:172.21.15.17:6789/0 conn(0x7fa9c00c9670 legacy=0x7fa9c016c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:43.308 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.327+0000 7fa9b6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:43.308 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.327+0000 7fa9b6ffd640 1 monclient: found mon.c 2024-03-29T11:19:43.308 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.327+0000 7fa9b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:43.308 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.327+0000 7fa9b6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:19:43.308 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.327+0000 7fa9b6ffd640 1 -- 172.21.15.17:0/394084113 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c00c4bd0 con 0x7fa9c0006e30 2024-03-29T11:19:43.308 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.327+0000 7fa9b6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:19:43.308 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.327+0000 7fa9b6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:43.308 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.327+0000 7fa9b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:43.308 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.327+0000 7fa9c759e640 5 monclient: authenticate success, global_id 4262 2024-03-29T11:19:43.309 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.327+0000 7fa9c759e640 10 monclient: _renew_subs 2024-03-29T11:19:43.309 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.327+0000 7fa9c759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:19:43.309 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.327+0000 7fa9c759e640 1 -- 172.21.15.17:0/394084113 --> v1:172.21.15.17:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9c00c4dc0 con 0x7fa9c0006e30 2024-03-29T11:19:43.309 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.328+0000 7fa9c759e640 10 client.4262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:43.309 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.328+0000 7fa9c759e640 10 monclient: _renew_subs 2024-03-29T11:19:43.309 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.328+0000 7fa9c759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:19:43.309 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.328+0000 7fa9c759e640 1 -- 172.21.15.17:0/394084113 --> v1:172.21.15.17:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9c002f3c0 con 0x7fa9c0006e30 2024-03-29T11:19:43.309 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.328+0000 7fa9c759e640 1 librados: init done 2024-03-29T11:19:43.310 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.330+0000 7fa9b4ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:43.310 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.330+0000 7fa9b4ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:43.310 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.330+0000 7fa9b4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:19:43.310 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.330+0000 7fa9b4ff9640 1 -- 172.21.15.17:0/394084113 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9c0007b90 con 0x7fa9c0006e30 2024-03-29T11:19:43.312 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.335+0000 7fa9b6ffd640 1 -- 172.21.15.17:0/394084113 <== mon.2 v1:172.21.15.17:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9bc004a90 con 0x7fa9c0006e30 2024-03-29T11:19:43.312 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.335+0000 7fa9b6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:43.312 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.335+0000 7fa9b6ffd640 1 -- 172.21.15.17:0/394084113 <== mon.2 v1:172.21.15.17:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fa9bc004e60 con 0x7fa9c0006e30 2024-03-29T11:19:43.312 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.335+0000 7fa9b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:43.313 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.335+0000 7fa9b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:43.313 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.335+0000 7fa9b6ffd640 10 monclient: dump: 2024-03-29T11:19:43.313 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:19:43.313 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:19:43.313 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:43.313 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:43.313 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:43.313 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:43.313 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:19:43.313 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:19:43.313 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:19:43.313 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:43.313 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.335+0000 7fa9b7fff640 4 set_mon_vals no callback set 2024-03-29T11:19:43.313 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.336+0000 7fa9b6ffd640 1 -- 172.21.15.17:0/394084113 <== mon.2 v1:172.21.15.17:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7fa9bc005080 con 0x7fa9c0006e30 2024-03-29T11:19:43.314 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.336+0000 7fa9c5d9b640 10 monclient: get_auth_request con 0x7fa9a806dcd0 auth_method 0 2024-03-29T11:19:43.314 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.337+0000 7fa9b6ffd640 1 -- 172.21.15.17:0/394084113 <== mon.2 v1:172.21.15.17:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7fa9bc084b70 con 0x7fa9c0006e30 2024-03-29T11:19:43.315 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.337+0000 7fa9b6ffd640 10 client.4262.objecter ms_dispatch 0x7fa9c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:43.315 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.337+0000 7fa9b6ffd640 3 client.4262.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:43.315 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.337+0000 7fa9b6ffd640 3 client.4262.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:43.315 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.337+0000 7fa9b6ffd640 20 client.4262.objecter dump_active .. 0 homeless 2024-03-29T11:19:43.315 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.337+0000 7fa9b6ffd640 10 client.4262.objecter ms_handle_connect 0x7fa9a806dcd0 2024-03-29T11:19:43.322 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.342+0000 7fa9b6ffd640 1 -- 172.21.15.17:0/394084113 <== mon.2 v1:172.21.15.17:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa9bc0505d0 con 0x7fa9c0006e30 2024-03-29T11:19:43.322 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.342+0000 7fa9b6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:43.322 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.342+0000 7fa9b6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:43.648 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.671+0000 7fa9b4ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:43.648 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.671+0000 7fa9b4ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:43.648 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.671+0000 7fa9b4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:19:43.648 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.671+0000 7fa9b4ff9640 1 -- 172.21.15.17:0/394084113 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9c00cdae0 con 0x7fa9c0006e30 2024-03-29T11:19:43.652 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.675+0000 7fa9b6ffd640 1 -- 172.21.15.17:0/394084113 <== mon.2 v1:172.21.15.17:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7fa9bc052500 con 0x7fa9c0006e30 2024-03-29T11:19:43.652 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.675+0000 7fa9b6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:43.652 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.675+0000 7fa9b6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:43.655 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:19:43.655 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:43.658 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.681+0000 7fa9b4ff9640 10 librados: watch_flush enter 2024-03-29T11:19:43.658 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.682+0000 7fa9b4ff9640 10 librados: watch_flush exit 2024-03-29T11:19:43.658 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.682+0000 7fa9b4ff9640 20 client.4262.objecter shutdown clearing up homeless session... 2024-03-29T11:19:43.658 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.682+0000 7fa9b4ff9640 10 client.4262.objecter successfully canceled tick 2024-03-29T11:19:43.659 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.682+0000 7fa9b4ff9640 1 -- 172.21.15.17:0/394084113 >> v1:172.21.15.17:6816/3236185052 conn(0x7fa9a806dcd0 legacy=0x7fa9a8070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:43.659 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.682+0000 7fa9b4ff9640 10 monclient: shutdown 2024-03-29T11:19:43.659 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.682+0000 7fa9b4ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:43.659 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.682+0000 7fa9b4ff9640 1 -- 172.21.15.17:0/394084113 >> v1:172.21.15.17:6790/0 conn(0x7fa9c0006e30 legacy=0x7fa9c0160cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:43.659 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.682+0000 7fa9b4ff9640 1 -- 172.21.15.17:0/394084113 shutdown_connections 2024-03-29T11:19:43.659 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.682+0000 7fa9b4ff9640 1 -- 172.21.15.17:0/394084113 >> 172.21.15.17:0/394084113 conn(0x7fa9c0005c00 msgr2=0x7fa9c0005fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:43.660 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.683+0000 7fa9b4ff9640 1 -- 172.21.15.17:0/394084113 shutdown_connections 2024-03-29T11:19:43.660 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.683+0000 7fa9b4ff9640 1 -- 172.21.15.17:0/394084113 wait complete. 2024-03-29T11:19:43.660 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:43.683+0000 7fa9b4ff9640 1 librados: shutdown 2024-03-29T11:19:43.681 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:50.487 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:50.636 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.659+0000 7f5c9633d640 10 monclient: get_monmap_and_config 2024-03-29T11:19:50.636 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.659+0000 7f5c9633d640 10 monclient: build_initial_monmap 2024-03-29T11:19:50.636 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.659+0000 7f5c9633d640 10 monclient: monmap: 2024-03-29T11:19:50.636 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:19:50.636 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:19:50.637 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:19:50.660622+0000 2024-03-29T11:19:50.637 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:19:50.660622+0000 2024-03-29T11:19:50.637 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:50.637 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:50.637 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:19:50.637 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:19:50.637 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:19:50.637 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:50.641 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.664+0000 7f5c9633d640 1 Processor -- start 2024-03-29T11:19:50.642 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.664+0000 7f5c9633d640 1 -- start start 2024-03-29T11:19:50.642 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.664+0000 7f5c9633d640 10 monclient: init 2024-03-29T11:19:50.642 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.665+0000 7f5c9633d640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:50.642 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.665+0000 7f5c9633d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:50.642 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.665+0000 7f5c9633d640 10 monclient(hunting): picked mon.noname-c con 0x7f5c88038c60 addr v1:172.21.15.94:6789/0 2024-03-29T11:19:50.642 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.665+0000 7f5c9633d640 10 monclient(hunting): picked mon.noname-b con 0x7f5c880324b0 addr v1:172.21.15.17:6790/0 2024-03-29T11:19:50.643 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.667+0000 7f5c9633d640 10 monclient(hunting): picked mon.noname-a con 0x7f5c880d11e0 addr v1:172.21.15.17:6789/0 2024-03-29T11:19:50.644 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.667+0000 7f5c9633d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c880caf80 con 0x7f5c880d11e0 2024-03-29T11:19:50.644 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.667+0000 7f5c9633d640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c88039850 con 0x7f5c88038c60 2024-03-29T11:19:50.644 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.667+0000 7f5c9633d640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c880c78d0 con 0x7f5c880324b0 2024-03-29T11:19:50.644 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.667+0000 7f5c9633d640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:50.644 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.667+0000 7f5c9633d640 10 monclient(hunting): authenticate will time out at 896.872038s 2024-03-29T11:19:50.648 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.671+0000 7f5c948da640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f5c880d11e0 0x7f5c880d35d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:34615/0 (socket says 172.21.15.17:34615) 2024-03-29T11:19:50.648 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.671+0000 7f5c948da640 1 -- 172.21.15.17:0/770689597 learned_addr learned my addr 172.21.15.17:0/770689597 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:19:50.649 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.672+0000 7f5c8effd640 1 -- 172.21.15.17:0/770689597 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4137088265 0 0) 0x7f5c880caf80 con 0x7f5c880d11e0 2024-03-29T11:19:50.649 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.672+0000 7f5c8effd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:50.649 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.672+0000 7f5c8effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:50.649 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.672+0000 7f5c8effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:50.649 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.672+0000 7f5c8effd640 10 monclient(hunting): my global_id is 4248 2024-03-29T11:19:50.649 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.672+0000 7f5c8effd640 1 -- 172.21.15.17:0/770689597 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c80003520 con 0x7f5c880d11e0 2024-03-29T11:19:50.653 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.675+0000 7f5c8effd640 1 -- 172.21.15.17:0/770689597 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 474328714 0 0) 0x7f5c80003520 con 0x7f5c880d11e0 2024-03-29T11:19:50.653 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.675+0000 7f5c8effd640 1 -- 172.21.15.17:0/770689597 >> v1:172.21.15.17:6790/0 conn(0x7f5c880324b0 legacy=0x7f5c880cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:19:50.653 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.675+0000 7f5c8effd640 1 -- 172.21.15.17:0/770689597 >> v1:172.21.15.94:6789/0 conn(0x7f5c88038c60 legacy=0x7f5c880c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:50.653 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.675+0000 7f5c8effd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:50.653 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.675+0000 7f5c8effd640 1 monclient: found mon.noname-a 2024-03-29T11:19:50.653 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.675+0000 7f5c8effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:50.653 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.675+0000 7f5c8effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.17:6789/0 2024-03-29T11:19:50.653 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.675+0000 7f5c8effd640 1 -- 172.21.15.17:0/770689597 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c880c7a60 con 0x7f5c880d11e0 2024-03-29T11:19:50.654 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.676+0000 7f5c8effd640 10 monclient: _finish_auth 0 2024-03-29T11:19:50.654 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.676+0000 7f5c8effd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:50.654 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.676+0000 7f5c8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:50.654 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.676+0000 7f5c8effd640 1 -- 172.21.15.17:0/770689597 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f5c9004f730 con 0x7f5c880d11e0 2024-03-29T11:19:50.654 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.676+0000 7f5c8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:50.654 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.676+0000 7f5c8effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:50.654 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.676+0000 7f5c8effd640 10 monclient: dump: 2024-03-29T11:19:50.654 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:19:50.654 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:19:50.654 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:50.654 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:50.654 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:50.654 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:50.654 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:19:50.654 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:19:50.655 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:19:50.655 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:50.655 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.676+0000 7f5c9633d640 5 monclient: authenticate success, global_id 4248 2024-03-29T11:19:50.655 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.676+0000 7f5c9633d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:50.658 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.678+0000 7f5c8effd640 1 -- 172.21.15.17:0/770689597 <== mon.0 v1:172.21.15.17:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c90075f10 con 0x7f5c880d11e0 2024-03-29T11:19:50.659 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.678+0000 7f5c8effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:50.659 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.678+0000 7f5c8effd640 1 -- 172.21.15.17:0/770689597 <== mon.0 v1:172.21.15.17:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f5c900758b0 con 0x7f5c880d11e0 2024-03-29T11:19:50.659 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.678+0000 7f5c8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:50.659 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.678+0000 7f5c8effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:50.659 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.678+0000 7f5c8effd640 10 monclient: dump: 2024-03-29T11:19:50.659 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:19:50.659 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:19:50.659 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:50.659 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:50.659 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:50.659 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:50.659 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:19:50.659 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:19:50.659 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:19:50.659 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:50.659 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.680+0000 7f5c9633d640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:50.660 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.680+0000 7f5c9633d640 4 set_mon_vals no callback set 2024-03-29T11:19:50.660 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.681+0000 7f5c9633d640 10 monclient: shutdown 2024-03-29T11:19:50.660 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.681+0000 7f5c9633d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:50.660 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.681+0000 7f5c9633d640 1 -- 172.21.15.17:0/770689597 >> v1:172.21.15.17:6789/0 conn(0x7f5c880d11e0 legacy=0x7f5c880d35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:50.660 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.682+0000 7f5c9633d640 1 -- 172.21.15.17:0/770689597 shutdown_connections 2024-03-29T11:19:50.660 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.682+0000 7f5c9633d640 1 -- 172.21.15.17:0/770689597 >> 172.21.15.17:0/770689597 conn(0x7f5c88005c20 msgr2=0x7f5c88006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:50.660 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.682+0000 7f5c9633d640 1 -- 172.21.15.17:0/770689597 shutdown_connections 2024-03-29T11:19:50.660 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.682+0000 7f5c9633d640 1 -- 172.21.15.17:0/770689597 wait complete. 2024-03-29T11:19:50.661 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.683+0000 7f5c9633d640 10 monclient: build_initial_monmap 2024-03-29T11:19:50.661 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.683+0000 7f5c9633d640 10 monclient: monmap: 2024-03-29T11:19:50.661 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:19:50.661 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:50.661 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:19:50.661 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:19:50.661 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:50.661 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:50.661 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:19:50.661 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:19:50.661 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:19:50.661 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:50.661 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.683+0000 7f5c9633d640 1 librados: starting msgr at 2024-03-29T11:19:50.662 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.683+0000 7f5c9633d640 1 librados: starting objecter 2024-03-29T11:19:50.662 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.683+0000 7f5c9633d640 1 Processor -- start 2024-03-29T11:19:50.663 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.684+0000 7f5c9633d640 1 -- start start 2024-03-29T11:19:50.663 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.684+0000 7f5c9633d640 1 librados: setting wanted keys 2024-03-29T11:19:50.663 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.684+0000 7f5c9633d640 1 librados: calling monclient init 2024-03-29T11:19:50.663 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.684+0000 7f5c9633d640 10 monclient: init 2024-03-29T11:19:50.663 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.684+0000 7f5c9633d640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:50.663 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.684+0000 7f5c9633d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:50.663 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.684+0000 7f5c9633d640 10 monclient(hunting): picked mon.noname-a con 0x7f5c880324b0 addr v1:172.21.15.17:6789/0 2024-03-29T11:19:50.663 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.684+0000 7f5c9633d640 10 monclient(hunting): picked mon.noname-c con 0x7f5c88038c60 addr v1:172.21.15.17:6790/0 2024-03-29T11:19:50.665 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.688+0000 7f5c9633d640 10 monclient(hunting): picked mon.noname-b con 0x7f5c880d11e0 addr v1:172.21.15.94:6789/0 2024-03-29T11:19:50.665 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.688+0000 7f5c9633d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c880c3d10 con 0x7f5c880324b0 2024-03-29T11:19:50.665 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.688+0000 7f5c9633d640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c88160ac0 con 0x7f5c880d11e0 2024-03-29T11:19:50.665 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.688+0000 7f5c9633d640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c8802f600 con 0x7f5c88038c60 2024-03-29T11:19:50.665 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.689+0000 7f5c9633d640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:50.665 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.689+0000 7f5c9633d640 10 monclient(hunting): authenticate will time out at 896.893963s 2024-03-29T11:19:50.667 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.689+0000 7f5c8ffff640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f5c880324b0 0x7f5c880c27d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:34741/0 (socket says 172.21.15.17:34741) 2024-03-29T11:19:50.667 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.689+0000 7f5c8ffff640 1 -- 172.21.15.17:0/1505564040 learned_addr learned my addr 172.21.15.17:0/1505564040 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:19:50.667 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.689+0000 7f5c8cff9640 10 client.?.objecter ms_handle_connect 0x7f5c880324b0 2024-03-29T11:19:50.667 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.689+0000 7f5c8cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:50.667 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.689+0000 7f5c8cff9640 1 -- 172.21.15.17:0/1505564040 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2823810287 0 0) 0x7f5c880c3d10 con 0x7f5c880324b0 2024-03-29T11:19:50.667 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.690+0000 7f5c8cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:50.667 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.690+0000 7f5c8cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:50.667 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.690+0000 7f5c8cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:50.667 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.690+0000 7f5c8cff9640 10 monclient(hunting): my global_id is 4251 2024-03-29T11:19:50.667 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.690+0000 7f5c8cff9640 1 -- 172.21.15.17:0/1505564040 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c74004eb0 con 0x7f5c880324b0 2024-03-29T11:19:50.667 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.690+0000 7f5c8cff9640 1 -- 172.21.15.17:0/1505564040 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1833082246 0 0) 0x7f5c74004eb0 con 0x7f5c880324b0 2024-03-29T11:19:50.668 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.690+0000 7f5c8cff9640 1 -- 172.21.15.17:0/1505564040 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c880c3d10 con 0x7f5c880324b0 2024-03-29T11:19:50.669 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.691+0000 7f5c8cff9640 10 client.?.objecter ms_handle_connect 0x7f5c880d11e0 2024-03-29T11:19:50.669 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.691+0000 7f5c8cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:50.669 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.691+0000 7f5c8cff9640 1 -- 172.21.15.17:0/1505564040 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f5c7c001940 con 0x7f5c880324b0 2024-03-29T11:19:50.669 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.691+0000 7f5c8cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:50.669 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.691+0000 7f5c8cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:50.669 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.691+0000 7f5c8cff9640 10 monclient(hunting): dump: 2024-03-29T11:19:50.669 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:19:50.669 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:19:50.669 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:50.669 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:50.669 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:50.669 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:50.669 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:19:50.670 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:19:50.670 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:19:50.670 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:50.670 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.691+0000 7f5c8cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:50.670 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.692+0000 7f5c8cff9640 1 -- 172.21.15.17:0/1505564040 <== mon.0 v1:172.21.15.17:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4239038815 0 0) 0x7f5c880c3d10 con 0x7f5c880324b0 2024-03-29T11:19:50.670 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.692+0000 7f5c8cff9640 1 -- 172.21.15.17:0/1505564040 >> v1:172.21.15.17:6790/0 conn(0x7f5c88038c60 legacy=0x7f5c88160fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:19:50.673 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.695+0000 7f5c8cff9640 1 -- 172.21.15.17:0/1505564040 >> v1:172.21.15.94:6789/0 conn(0x7f5c880d11e0 legacy=0x7f5c880c4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:50.673 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.695+0000 7f5c8cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:19:50.673 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.695+0000 7f5c8cff9640 1 monclient: found mon.a 2024-03-29T11:19:50.673 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.695+0000 7f5c8cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:50.673 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.695+0000 7f5c8cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:19:50.673 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.695+0000 7f5c8cff9640 1 -- 172.21.15.17:0/1505564040 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c881616b0 con 0x7f5c880324b0 2024-03-29T11:19:50.673 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.695+0000 7f5c8cff9640 10 monclient: _finish_auth 0 2024-03-29T11:19:50.673 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.695+0000 7f5c8cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:19:50.674 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.695+0000 7f5c8cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:50.674 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.695+0000 7f5c9633d640 5 monclient: authenticate success, global_id 4251 2024-03-29T11:19:50.674 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.695+0000 7f5c9633d640 10 monclient: _renew_subs 2024-03-29T11:19:50.674 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.695+0000 7f5c9633d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:19:50.674 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.695+0000 7f5c9633d640 1 -- 172.21.15.17:0/1505564040 --> v1:172.21.15.17:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c8802fde0 con 0x7f5c880324b0 2024-03-29T11:19:50.674 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.695+0000 7f5c9633d640 10 client.4251.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:50.674 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.695+0000 7f5c9633d640 10 monclient: _renew_subs 2024-03-29T11:19:50.674 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.695+0000 7f5c9633d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:19:50.674 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.695+0000 7f5c9633d640 1 -- 172.21.15.17:0/1505564040 --> v1:172.21.15.17:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c880c45f0 con 0x7f5c880324b0 2024-03-29T11:19:50.674 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.695+0000 7f5c9633d640 1 librados: init done 2024-03-29T11:19:50.674 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.696+0000 7f5c8cff9640 1 -- 172.21.15.17:0/1505564040 <== mon.0 v1:172.21.15.17:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c7c001aa0 con 0x7f5c880324b0 2024-03-29T11:19:50.674 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.696+0000 7f5c8cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:50.674 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.696+0000 7f5c8cff9640 1 -- 172.21.15.17:0/1505564040 <== mon.0 v1:172.21.15.17:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f5c7c003a60 con 0x7f5c880324b0 2024-03-29T11:19:50.674 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.696+0000 7f5c8cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:50.674 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.696+0000 7f5c8cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:50.675 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.696+0000 7f5c8cff9640 10 monclient: dump: 2024-03-29T11:19:50.675 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:19:50.675 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:19:50.675 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:50.675 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:50.675 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:50.675 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:50.675 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:19:50.675 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:19:50.675 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:19:50.675 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:50.675 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.696+0000 7f5c8e7fc640 4 set_mon_vals no callback set 2024-03-29T11:19:50.676 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.698+0000 7f5c8cff9640 1 -- 172.21.15.17:0/1505564040 <== mon.0 v1:172.21.15.17:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f5c7c003bc0 con 0x7f5c880324b0 2024-03-29T11:19:50.676 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.698+0000 7f5c8cff9640 1 -- 172.21.15.17:0/1505564040 <== mon.0 v1:172.21.15.17:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f5c7c084a30 con 0x7f5c880324b0 2024-03-29T11:19:50.676 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.698+0000 7f5c8cff9640 10 client.4251.objecter ms_dispatch 0x7f5c90000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:50.676 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.698+0000 7f5c8cff9640 3 client.4251.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:50.676 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.698+0000 7f5c8cff9640 3 client.4251.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:50.677 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.699+0000 7f5c8cff9640 20 client.4251.objecter dump_active .. 0 homeless 2024-03-29T11:19:50.677 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.699+0000 7f5c8f7fe640 10 monclient: get_auth_request con 0x7f5c7406db50 auth_method 0 2024-03-29T11:19:50.677 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.699+0000 7f5c9633d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:50.677 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.699+0000 7f5c9633d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:50.677 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.699+0000 7f5c9633d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:19:50.677 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.699+0000 7f5c9633d640 1 -- 172.21.15.17:0/1505564040 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c70005080 con 0x7f5c880324b0 2024-03-29T11:19:50.677 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.700+0000 7f5c8cff9640 10 client.4251.objecter ms_handle_connect 0x7f5c7406db50 2024-03-29T11:19:50.689 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.713+0000 7f5c8cff9640 1 -- 172.21.15.17:0/1505564040 <== mon.0 v1:172.21.15.17:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5c7c050310 con 0x7f5c880324b0 2024-03-29T11:19:50.689 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.713+0000 7f5c8cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:50.690 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:50.713+0000 7f5c8cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:51.020 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:51.044+0000 7f5c9633d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:51.021 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:51.044+0000 7f5c9633d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:51.021 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:51.044+0000 7f5c9633d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:19:51.021 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:51.044+0000 7f5c9633d640 1 -- 172.21.15.17:0/1505564040 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5c70005330 con 0x7f5c880324b0 2024-03-29T11:19:51.023 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:51.047+0000 7f5c8cff9640 1 -- 172.21.15.17:0/1505564040 <== mon.0 v1:172.21.15.17:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f5c7c052240 con 0x7f5c880324b0 2024-03-29T11:19:51.024 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:51.047+0000 7f5c8cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:51.024 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:51.047+0000 7f5c8cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:51.029 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:19:51.029 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:51.030 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:51.054+0000 7f5c9633d640 10 librados: watch_flush enter 2024-03-29T11:19:51.031 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:51.054+0000 7f5c9633d640 10 librados: watch_flush exit 2024-03-29T11:19:51.031 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:51.054+0000 7f5c9633d640 20 client.4251.objecter shutdown clearing up homeless session... 2024-03-29T11:19:51.031 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:51.054+0000 7f5c9633d640 10 client.4251.objecter successfully canceled tick 2024-03-29T11:19:51.032 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:51.056+0000 7f5c9633d640 1 -- 172.21.15.17:0/1505564040 >> v1:172.21.15.17:6816/3236185052 conn(0x7f5c7406db50 legacy=0x7f5c74070010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:51.033 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:51.056+0000 7f5c9633d640 10 monclient: shutdown 2024-03-29T11:19:51.033 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:51.056+0000 7f5c9633d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:51.033 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:51.056+0000 7f5c9633d640 1 -- 172.21.15.17:0/1505564040 >> v1:172.21.15.17:6789/0 conn(0x7f5c880324b0 legacy=0x7f5c880c27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:51.033 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:51.056+0000 7f5c9633d640 1 -- 172.21.15.17:0/1505564040 shutdown_connections 2024-03-29T11:19:51.033 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:51.056+0000 7f5c9633d640 1 -- 172.21.15.17:0/1505564040 >> 172.21.15.17:0/1505564040 conn(0x7f5c88005c20 msgr2=0x7f5c8803ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:51.033 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:51.056+0000 7f5c9633d640 1 -- 172.21.15.17:0/1505564040 shutdown_connections 2024-03-29T11:19:51.036 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:51.059+0000 7f5c9633d640 1 -- 172.21.15.17:0/1505564040 wait complete. 2024-03-29T11:19:51.036 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:51.059+0000 7f5c9633d640 1 librados: shutdown 2024-03-29T11:19:51.053 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:57.860 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:58.014 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.036+0000 7fee7ff15640 10 monclient: get_monmap_and_config 2024-03-29T11:19:58.014 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.036+0000 7fee7ff15640 10 monclient: build_initial_monmap 2024-03-29T11:19:58.014 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.036+0000 7fee7ff15640 10 monclient: monmap: 2024-03-29T11:19:58.014 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:19:58.014 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:19:58.014 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:19:58.037397+0000 2024-03-29T11:19:58.014 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:19:58.037397+0000 2024-03-29T11:19:58.014 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:58.015 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:58.015 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:19:58.015 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:19:58.015 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:19:58.015 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:58.015 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.037+0000 7fee7ff15640 1 Processor -- start 2024-03-29T11:19:58.015 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.037+0000 7fee7ff15640 1 -- start start 2024-03-29T11:19:58.015 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.037+0000 7fee7ff15640 10 monclient: init 2024-03-29T11:19:58.015 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.038+0000 7fee7ff15640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:58.016 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.038+0000 7fee7ff15640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:58.016 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.039+0000 7fee7ff15640 10 monclient(hunting): picked mon.noname-a con 0x7fee780a9ac0 addr v1:172.21.15.17:6789/0 2024-03-29T11:19:58.016 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.039+0000 7fee7ff15640 10 monclient(hunting): picked mon.noname-c con 0x7fee7813ee90 addr v1:172.21.15.94:6789/0 2024-03-29T11:19:58.016 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.039+0000 7fee7ff15640 10 monclient(hunting): picked mon.noname-b con 0x7fee780af900 addr v1:172.21.15.17:6790/0 2024-03-29T11:19:58.016 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.039+0000 7fee7ff15640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee780afd10 con 0x7fee780a9ac0 2024-03-29T11:19:58.016 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.039+0000 7fee7ff15640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee780af6f0 con 0x7fee7813ee90 2024-03-29T11:19:58.016 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.039+0000 7fee7ff15640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee780aac80 con 0x7fee780af900 2024-03-29T11:19:58.016 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.039+0000 7fee7ff15640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:58.016 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.039+0000 7fee7ff15640 10 monclient(hunting): authenticate will time out at 904.244317s 2024-03-29T11:19:58.024 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.047+0000 7fee7dcb1640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fee780a9ac0 0x7fee780adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:42747/0 (socket says 172.21.15.17:42747) 2024-03-29T11:19:58.024 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.047+0000 7fee7dcb1640 1 -- 172.21.15.17:0/2350135834 learned_addr learned my addr 172.21.15.17:0/2350135834 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:19:58.024 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.048+0000 7fee7ccaf640 1 -- 172.21.15.17:0/2350135834 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1487902205 0 0) 0x7fee780afd10 con 0x7fee780a9ac0 2024-03-29T11:19:58.025 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.048+0000 7fee7ccaf640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.025 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.048+0000 7fee7ccaf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.025 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.048+0000 7fee7ccaf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.025 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.048+0000 7fee7ccaf640 10 monclient(hunting): my global_id is 4254 2024-03-29T11:19:58.025 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.048+0000 7fee7ccaf640 1 -- 172.21.15.17:0/2350135834 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee60003520 con 0x7fee780a9ac0 2024-03-29T11:19:58.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.048+0000 7fee7ccaf640 1 -- 172.21.15.17:0/2350135834 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2772072047 0 0) 0x7fee60003520 con 0x7fee780a9ac0 2024-03-29T11:19:58.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.048+0000 7fee7ccaf640 1 -- 172.21.15.17:0/2350135834 >> v1:172.21.15.17:6790/0 conn(0x7fee780af900 legacy=0x7fee78149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:19:58.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.048+0000 7fee7ccaf640 1 -- 172.21.15.17:0/2350135834 >> v1:172.21.15.94:6789/0 conn(0x7fee7813ee90 legacy=0x7fee7813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:19:58.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.048+0000 7fee7ccaf640 10 monclient: _finish_hunting 0 2024-03-29T11:19:58.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.048+0000 7fee7ccaf640 1 monclient: found mon.noname-a 2024-03-29T11:19:58.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.048+0000 7fee7ccaf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:58.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.048+0000 7fee7ccaf640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.17:6789/0 2024-03-29T11:19:58.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.048+0000 7fee7ccaf640 1 -- 172.21.15.17:0/2350135834 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee780aaf60 con 0x7fee780a9ac0 2024-03-29T11:19:58.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.048+0000 7fee7ccaf640 10 monclient: _finish_auth 0 2024-03-29T11:19:58.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.049+0000 7fee7ccaf640 10 monclient: _check_auth_tickets 2024-03-29T11:19:58.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.049+0000 7fee7ccaf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:58.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.049+0000 7fee7ccaf640 1 -- 172.21.15.17:0/2350135834 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fee68001a10 con 0x7fee780a9ac0 2024-03-29T11:19:58.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.049+0000 7fee7ccaf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.027 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.049+0000 7fee7ccaf640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:58.027 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.049+0000 7fee7ccaf640 10 monclient: dump: 2024-03-29T11:19:58.027 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:19:58.027 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:19:58.027 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:58.027 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:58.027 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.027 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:58.027 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:19:58.027 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:19:58.027 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:19:58.027 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:58.027 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.049+0000 7fee7ccaf640 1 -- 172.21.15.17:0/2350135834 <== mon.0 v1:172.21.15.17:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee68001ca0 con 0x7fee780a9ac0 2024-03-29T11:19:58.027 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.049+0000 7fee7ccaf640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:58.027 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.049+0000 7fee7ccaf640 1 -- 172.21.15.17:0/2350135834 <== mon.0 v1:172.21.15.17:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fee68002030 con 0x7fee780a9ac0 2024-03-29T11:19:58.027 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.049+0000 7fee7ccaf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.028 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.049+0000 7fee7ccaf640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:58.028 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.049+0000 7fee7ccaf640 10 monclient: dump: 2024-03-29T11:19:58.028 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:19:58.028 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:19:58.028 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:58.028 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:58.028 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.028 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:58.028 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:19:58.028 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:19:58.028 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:19:58.028 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:58.029 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.051+0000 7fee7ff15640 5 monclient: authenticate success, global_id 4254 2024-03-29T11:19:58.029 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.051+0000 7fee7ff15640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:58.029 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.051+0000 7fee7ff15640 4 set_mon_vals no callback set 2024-03-29T11:19:58.029 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.051+0000 7fee7ff15640 10 monclient: shutdown 2024-03-29T11:19:58.029 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.052+0000 7fee7ff15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:58.029 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.052+0000 7fee7ff15640 1 -- 172.21.15.17:0/2350135834 >> v1:172.21.15.17:6789/0 conn(0x7fee780a9ac0 legacy=0x7fee780adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.029 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.052+0000 7fee7ff15640 1 -- 172.21.15.17:0/2350135834 shutdown_connections 2024-03-29T11:19:58.029 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.052+0000 7fee7ff15640 1 -- 172.21.15.17:0/2350135834 >> 172.21.15.17:0/2350135834 conn(0x7fee780b35e0 msgr2=0x7fee780b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:58.029 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.052+0000 7fee7ff15640 1 -- 172.21.15.17:0/2350135834 shutdown_connections 2024-03-29T11:19:58.030 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.053+0000 7fee7ff15640 1 -- 172.21.15.17:0/2350135834 wait complete. 2024-03-29T11:19:58.030 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.054+0000 7fee7ff15640 10 monclient: build_initial_monmap 2024-03-29T11:19:58.030 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.054+0000 7fee7ff15640 10 monclient: monmap: 2024-03-29T11:19:58.030 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:19:58.030 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:58.030 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:19:58.030 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:19:58.031 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:58.031 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:58.031 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:19:58.031 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:19:58.031 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:19:58.031 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:58.031 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.054+0000 7fee7ff15640 1 librados: starting msgr at 2024-03-29T11:19:58.031 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.054+0000 7fee7ff15640 1 librados: starting objecter 2024-03-29T11:19:58.031 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.055+0000 7fee7ff15640 1 Processor -- start 2024-03-29T11:19:58.034 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.057+0000 7fee7ff15640 1 -- start start 2024-03-29T11:19:58.034 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.057+0000 7fee7ff15640 1 librados: setting wanted keys 2024-03-29T11:19:58.034 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.057+0000 7fee7ff15640 1 librados: calling monclient init 2024-03-29T11:19:58.034 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.057+0000 7fee7ff15640 10 monclient: init 2024-03-29T11:19:58.035 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.058+0000 7fee7ff15640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:58.035 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.058+0000 7fee7ff15640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:58.035 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.058+0000 7fee7ff15640 10 monclient(hunting): picked mon.noname-b con 0x7fee780af900 addr v1:172.21.15.94:6789/0 2024-03-29T11:19:58.035 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.058+0000 7fee7ff15640 10 monclient(hunting): picked mon.noname-c con 0x7fee7813ee90 addr v1:172.21.15.17:6790/0 2024-03-29T11:19:58.035 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.058+0000 7fee7ff15640 10 monclient(hunting): picked mon.noname-a con 0x7fee78139a20 addr v1:172.21.15.17:6789/0 2024-03-29T11:19:58.035 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.058+0000 7fee7ff15640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee781d4150 con 0x7fee78139a20 2024-03-29T11:19:58.035 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.058+0000 7fee7ff15640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee781d5020 con 0x7fee780af900 2024-03-29T11:19:58.035 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.058+0000 7fee7ff15640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee781d43f0 con 0x7fee7813ee90 2024-03-29T11:19:58.035 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.058+0000 7fee7ff15640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:58.035 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.058+0000 7fee7ff15640 10 monclient(hunting): authenticate will time out at 904.263070s 2024-03-29T11:19:58.038 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.062+0000 7fee7dcb1640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fee780af900 0x7fee781d4910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:51379/0 (socket says 172.21.15.17:51379) 2024-03-29T11:19:58.039 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.062+0000 7fee7dcb1640 1 -- 172.21.15.17:0/3308096519 learned_addr learned my addr 172.21.15.17:0/3308096519 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:19:58.039 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.062+0000 7fee767fc640 10 client.?.objecter ms_handle_connect 0x7fee780af900 2024-03-29T11:19:58.039 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.062+0000 7fee767fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:58.039 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.063+0000 7fee767fc640 10 client.?.objecter ms_handle_connect 0x7fee7813ee90 2024-03-29T11:19:58.040 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.063+0000 7fee767fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:58.040 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.063+0000 7fee767fc640 10 client.?.objecter ms_handle_connect 0x7fee78139a20 2024-03-29T11:19:58.040 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.063+0000 7fee767fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:58.041 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.064+0000 7fee767fc640 1 -- 172.21.15.17:0/3308096519 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 32018202 0 0) 0x7fee781d4150 con 0x7fee78139a20 2024-03-29T11:19:58.041 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.064+0000 7fee767fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.041 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.064+0000 7fee767fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.041 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.064+0000 7fee767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.041 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.064+0000 7fee767fc640 10 monclient(hunting): my global_id is 4257 2024-03-29T11:19:58.041 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.064+0000 7fee767fc640 1 -- 172.21.15.17:0/3308096519 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee64004eb0 con 0x7fee78139a20 2024-03-29T11:19:58.041 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.064+0000 7fee767fc640 1 -- 172.21.15.17:0/3308096519 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2295488811 0 0) 0x7fee781d43f0 con 0x7fee7813ee90 2024-03-29T11:19:58.041 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.064+0000 7fee767fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.041 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.064+0000 7fee767fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.041 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.064+0000 7fee767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.042 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.064+0000 7fee767fc640 10 monclient(hunting): my global_id is 4265 2024-03-29T11:19:58.042 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.064+0000 7fee767fc640 1 -- 172.21.15.17:0/3308096519 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee781d4150 con 0x7fee7813ee90 2024-03-29T11:19:58.044 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.067+0000 7fee767fc640 1 -- 172.21.15.17:0/3308096519 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2715843107 0 0) 0x7fee64004eb0 con 0x7fee78139a20 2024-03-29T11:19:58.044 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.067+0000 7fee767fc640 1 -- 172.21.15.17:0/3308096519 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee781d43f0 con 0x7fee78139a20 2024-03-29T11:19:58.044 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.067+0000 7fee767fc640 1 -- 172.21.15.17:0/3308096519 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fee700017f0 con 0x7fee78139a20 2024-03-29T11:19:58.044 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.067+0000 7fee767fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.044 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.067+0000 7fee767fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:58.044 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.067+0000 7fee767fc640 10 monclient(hunting): dump: 2024-03-29T11:19:58.044 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:19:58.044 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:19:58.044 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:58.045 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:58.045 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.045 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:58.045 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:19:58.045 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:19:58.045 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:19:58.045 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:58.045 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.067+0000 7fee767fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:58.048 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.068+0000 7fee767fc640 1 -- 172.21.15.17:0/3308096519 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3702165016 0 0) 0x7fee781d4150 con 0x7fee7813ee90 2024-03-29T11:19:58.048 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.068+0000 7fee767fc640 1 -- 172.21.15.17:0/3308096519 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee64004eb0 con 0x7fee7813ee90 2024-03-29T11:19:58.048 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.068+0000 7fee767fc640 1 -- 172.21.15.17:0/3308096519 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fee6c001810 con 0x7fee7813ee90 2024-03-29T11:19:58.048 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.068+0000 7fee767fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.048 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.068+0000 7fee767fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:58.048 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.068+0000 7fee767fc640 10 monclient(hunting): dump: 2024-03-29T11:19:58.048 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:19:58.048 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:19:58.048 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:58.048 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:58.048 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.048 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:58.048 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:19:58.049 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:19:58.049 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:19:58.049 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:58.049 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.068+0000 7fee767fc640 1 -- 172.21.15.17:0/3308096519 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1725521283 0 0) 0x7fee781d5020 con 0x7fee780af900 2024-03-29T11:19:58.049 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.068+0000 7fee767fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.049 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.068+0000 7fee767fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.049 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.068+0000 7fee767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.049 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.068+0000 7fee767fc640 10 monclient(hunting): my global_id is 4276 2024-03-29T11:19:58.049 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.068+0000 7fee767fc640 1 -- 172.21.15.17:0/3308096519 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee781d4150 con 0x7fee780af900 2024-03-29T11:19:58.049 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.069+0000 7fee767fc640 1 -- 172.21.15.17:0/3308096519 <== mon.2 v1:172.21.15.17:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2084946598 0 0) 0x7fee64004eb0 con 0x7fee7813ee90 2024-03-29T11:19:58.049 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.069+0000 7fee767fc640 1 -- 172.21.15.17:0/3308096519 >> v1:172.21.15.94:6789/0 conn(0x7fee780af900 legacy=0x7fee781d4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.049 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.069+0000 7fee767fc640 1 -- 172.21.15.17:0/3308096519 >> v1:172.21.15.17:6789/0 conn(0x7fee78139a20 legacy=0x7fee78137840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.049 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.069+0000 7fee767fc640 10 monclient: _finish_hunting 0 2024-03-29T11:19:58.049 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.069+0000 7fee767fc640 1 monclient: found mon.c 2024-03-29T11:19:58.050 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.069+0000 7fee767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:58.050 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.069+0000 7fee767fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:19:58.050 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.069+0000 7fee767fc640 1 -- 172.21.15.17:0/3308096519 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee78141750 con 0x7fee7813ee90 2024-03-29T11:19:58.050 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.069+0000 7fee767fc640 10 monclient: _finish_auth 0 2024-03-29T11:19:58.050 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.069+0000 7fee767fc640 10 monclient: _check_auth_tickets 2024-03-29T11:19:58.050 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.069+0000 7fee767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:58.050 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.069+0000 7fee767fc640 1 -- 172.21.15.17:0/3308096519 <== mon.2 v1:172.21.15.17:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee6c001970 con 0x7fee7813ee90 2024-03-29T11:19:58.050 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.069+0000 7fee767fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:58.050 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.069+0000 7fee77fff640 4 set_mon_vals no callback set 2024-03-29T11:19:58.050 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.069+0000 7fee767fc640 1 -- 172.21.15.17:0/3308096519 <== mon.2 v1:172.21.15.17:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fee6c003790 con 0x7fee7813ee90 2024-03-29T11:19:58.050 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.069+0000 7fee767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.050 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.069+0000 7fee767fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:58.050 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.069+0000 7fee767fc640 10 monclient: dump: 2024-03-29T11:19:58.050 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:19:58.051 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:19:58.051 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:58.051 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:19:58.051 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.051 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:19:58.051 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:19:58.051 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:19:58.051 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:19:58.051 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:19:58.051 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.070+0000 7fee7ff15640 5 monclient: authenticate success, global_id 4265 2024-03-29T11:19:58.051 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.070+0000 7fee7ff15640 10 monclient: _renew_subs 2024-03-29T11:19:58.051 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.070+0000 7fee7ff15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:19:58.051 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.070+0000 7fee7ff15640 1 -- 172.21.15.17:0/3308096519 --> v1:172.21.15.17:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee78141940 con 0x7fee7813ee90 2024-03-29T11:19:58.051 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.070+0000 7fee7ff15640 10 client.4265.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:58.051 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.070+0000 7fee7ff15640 10 monclient: _renew_subs 2024-03-29T11:19:58.051 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.070+0000 7fee7ff15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:19:58.052 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.070+0000 7fee7ff15640 1 -- 172.21.15.17:0/3308096519 --> v1:172.21.15.17:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fee78140680 con 0x7fee7813ee90 2024-03-29T11:19:58.052 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.070+0000 7fee7ff15640 1 librados: init done 2024-03-29T11:19:58.052 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.074+0000 7fee7ff15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:58.052 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.074+0000 7fee7ff15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:58.052 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.074+0000 7fee7ff15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:19:58.054 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.074+0000 7fee7ff15640 1 -- 172.21.15.17:0/3308096519 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee780ad0e0 con 0x7fee7813ee90 2024-03-29T11:19:58.055 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.078+0000 7fee767fc640 1 -- 172.21.15.17:0/3308096519 <== mon.2 v1:172.21.15.17:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7fee6c019760 con 0x7fee7813ee90 2024-03-29T11:19:58.055 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.079+0000 7fee7dcb1640 10 monclient: get_auth_request con 0x7fee640701f0 auth_method 0 2024-03-29T11:19:58.055 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.079+0000 7fee767fc640 1 -- 172.21.15.17:0/3308096519 <== mon.2 v1:172.21.15.17:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7fee6c083820 con 0x7fee7813ee90 2024-03-29T11:19:58.055 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.079+0000 7fee767fc640 10 client.4265.objecter ms_dispatch 0x7fee78000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:58.055 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.079+0000 7fee767fc640 3 client.4265.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:58.056 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.079+0000 7fee767fc640 3 client.4265.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:58.056 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.079+0000 7fee767fc640 20 client.4265.objecter dump_active .. 0 homeless 2024-03-29T11:19:58.056 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.079+0000 7fee767fc640 10 client.4265.objecter ms_handle_connect 0x7fee640701f0 2024-03-29T11:19:58.063 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.087+0000 7fee767fc640 1 -- 172.21.15.17:0/3308096519 <== mon.2 v1:172.21.15.17:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fee6c04ef30 con 0x7fee7813ee90 2024-03-29T11:19:58.063 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.087+0000 7fee767fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:58.064 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.087+0000 7fee767fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:58.392 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.416+0000 7fee7ff15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:58.393 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.416+0000 7fee7ff15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:58.393 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.416+0000 7fee7ff15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:19:58.393 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.416+0000 7fee7ff15640 1 -- 172.21.15.17:0/3308096519 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee781d4580 con 0x7fee7813ee90 2024-03-29T11:19:58.394 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.417+0000 7fee767fc640 1 -- 172.21.15.17:0/3308096519 <== mon.2 v1:172.21.15.17:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7fee6c051030 con 0x7fee7813ee90 2024-03-29T11:19:58.394 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.418+0000 7fee767fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:58.394 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.418+0000 7fee767fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:58.395 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:19:58.396 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:58.402 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.422+0000 7fee7ff15640 10 librados: watch_flush enter 2024-03-29T11:19:58.402 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.422+0000 7fee7ff15640 10 librados: watch_flush exit 2024-03-29T11:19:58.402 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.422+0000 7fee7ff15640 20 client.4265.objecter shutdown clearing up homeless session... 2024-03-29T11:19:58.402 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.422+0000 7fee7ff15640 10 client.4265.objecter successfully canceled tick 2024-03-29T11:19:58.402 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.423+0000 7fee7ff15640 1 -- 172.21.15.17:0/3308096519 >> v1:172.21.15.17:6816/3236185052 conn(0x7fee640701f0 legacy=0x7fee640726b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.402 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.423+0000 7fee7ff15640 10 monclient: shutdown 2024-03-29T11:19:58.402 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.423+0000 7fee7ff15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:58.402 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.423+0000 7fee7ff15640 1 -- 172.21.15.17:0/3308096519 >> v1:172.21.15.17:6790/0 conn(0x7fee7813ee90 legacy=0x7fee78141040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.403 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.426+0000 7fee7ff15640 1 -- 172.21.15.17:0/3308096519 shutdown_connections 2024-03-29T11:19:58.403 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.426+0000 7fee7ff15640 1 -- 172.21.15.17:0/3308096519 >> 172.21.15.17:0/3308096519 conn(0x7fee780b35e0 msgr2=0x7fee780adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:58.403 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.426+0000 7fee7ff15640 1 -- 172.21.15.17:0/3308096519 shutdown_connections 2024-03-29T11:19:58.403 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.426+0000 7fee7ff15640 1 -- 172.21.15.17:0/3308096519 wait complete. 2024-03-29T11:19:58.403 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:19:58.426+0000 7fee7ff15640 1 librados: shutdown 2024-03-29T11:19:58.419 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:05.225 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:05.368 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.391+0000 7fe9489a4640 10 monclient: get_monmap_and_config 2024-03-29T11:20:05.368 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.391+0000 7fe9489a4640 10 monclient: build_initial_monmap 2024-03-29T11:20:05.368 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.391+0000 7fe9489a4640 10 monclient: monmap: 2024-03-29T11:20:05.368 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:20:05.368 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:20:05.369 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:20:05.392723+0000 2024-03-29T11:20:05.369 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:20:05.392723+0000 2024-03-29T11:20:05.369 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:05.369 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:05.369 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:20:05.369 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:20:05.369 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:20:05.369 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:05.370 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.392+0000 7fe9489a4640 1 Processor -- start 2024-03-29T11:20:05.370 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.392+0000 7fe9489a4640 1 -- start start 2024-03-29T11:20:05.370 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.392+0000 7fe9489a4640 10 monclient: init 2024-03-29T11:20:05.370 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.393+0000 7fe9489a4640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:05.370 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.393+0000 7fe9489a4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:20:05.370 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.393+0000 7fe9489a4640 10 monclient(hunting): picked mon.noname-a con 0x7fe9441422b0 addr v1:172.21.15.17:6789/0 2024-03-29T11:20:05.370 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.393+0000 7fe9489a4640 10 monclient(hunting): picked mon.noname-b con 0x7fe944142690 addr v1:172.21.15.17:6790/0 2024-03-29T11:20:05.370 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.393+0000 7fe9489a4640 10 monclient(hunting): picked mon.noname-c con 0x7fe9440b17d0 addr v1:172.21.15.94:6789/0 2024-03-29T11:20:05.370 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.393+0000 7fe9489a4640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe944142c80 con 0x7fe9441422b0 2024-03-29T11:20:05.371 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.393+0000 7fe9489a4640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9440aae40 con 0x7fe9440b17d0 2024-03-29T11:20:05.371 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.393+0000 7fe9489a4640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe94413fbd0 con 0x7fe944142690 2024-03-29T11:20:05.371 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.393+0000 7fe9489a4640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:05.371 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.393+0000 7fe9489a4640 10 monclient(hunting): authenticate will time out at 911.598367s 2024-03-29T11:20:05.371 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.393+0000 7fe941d9b640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7fe944142690 0x7fe9440b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:34178/0 (socket says 172.21.15.17:34178) 2024-03-29T11:20:05.371 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.393+0000 7fe941d9b640 1 -- 172.21.15.17:0/2998963914 learned_addr learned my addr 172.21.15.17:0/2998963914 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:20:05.374 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.394+0000 7fe94159a640 1 -- 172.21.15.17:0/2998963914 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1853707768 0 0) 0x7fe94413fbd0 con 0x7fe944142690 2024-03-29T11:20:05.374 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.394+0000 7fe94159a640 10 monclient(hunting): _negotiate 2024-03-29T11:20:05.374 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.394+0000 7fe94159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:05.375 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.394+0000 7fe94159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:05.375 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.394+0000 7fe94159a640 10 monclient(hunting): my global_id is 4268 2024-03-29T11:20:05.375 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.394+0000 7fe94159a640 1 -- 172.21.15.17:0/2998963914 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe92c003520 con 0x7fe944142690 2024-03-29T11:20:05.375 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.394+0000 7fe94159a640 1 -- 172.21.15.17:0/2998963914 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 884683599 0 0) 0x7fe944142c80 con 0x7fe9441422b0 2024-03-29T11:20:05.375 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.394+0000 7fe94159a640 10 monclient(hunting): _negotiate 2024-03-29T11:20:05.375 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.395+0000 7fe94159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:05.375 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.395+0000 7fe94159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:05.375 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.395+0000 7fe94159a640 10 monclient(hunting): my global_id is 4260 2024-03-29T11:20:05.375 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.395+0000 7fe94159a640 1 -- 172.21.15.17:0/2998963914 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe94413fbd0 con 0x7fe9441422b0 2024-03-29T11:20:05.375 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.395+0000 7fe94159a640 1 -- 172.21.15.17:0/2998963914 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3669676070 0 0) 0x7fe92c003520 con 0x7fe944142690 2024-03-29T11:20:05.375 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.395+0000 7fe94159a640 1 -- 172.21.15.17:0/2998963914 >> v1:172.21.15.94:6789/0 conn(0x7fe9440b17d0 legacy=0x7fe9440b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:20:05.375 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.395+0000 7fe94159a640 1 -- 172.21.15.17:0/2998963914 >> v1:172.21.15.17:6789/0 conn(0x7fe9441422b0 legacy=0x7fe94413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:05.375 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.395+0000 7fe94159a640 10 monclient: _finish_hunting 0 2024-03-29T11:20:05.375 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.395+0000 7fe94159a640 1 monclient: found mon.noname-b 2024-03-29T11:20:05.376 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.395+0000 7fe94159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:05.376 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.395+0000 7fe94159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.17:6790/0 2024-03-29T11:20:05.376 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.395+0000 7fe94159a640 1 -- 172.21.15.17:0/2998963914 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe944151940 con 0x7fe944142690 2024-03-29T11:20:05.376 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.395+0000 7fe94159a640 10 monclient: _finish_auth 0 2024-03-29T11:20:05.376 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.395+0000 7fe94159a640 10 monclient: _check_auth_tickets 2024-03-29T11:20:05.376 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.395+0000 7fe94159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:05.376 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.395+0000 7fe94159a640 1 -- 172.21.15.17:0/2998963914 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fe938001690 con 0x7fe944142690 2024-03-29T11:20:05.376 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.395+0000 7fe94159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:05.376 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.395+0000 7fe94159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:05.376 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.395+0000 7fe94159a640 10 monclient: dump: 2024-03-29T11:20:05.376 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:20:05.376 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:20:05.376 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:05.376 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:05.377 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:05.377 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:05.377 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:20:05.377 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:20:05.377 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:20:05.377 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:05.377 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.395+0000 7fe9489a4640 5 monclient: authenticate success, global_id 4268 2024-03-29T11:20:05.377 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.395+0000 7fe9489a4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:05.377 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.395+0000 7fe94159a640 1 -- 172.21.15.17:0/2998963914 <== mon.2 v1:172.21.15.17:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe938001870 con 0x7fe944142690 2024-03-29T11:20:05.377 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.395+0000 7fe94159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:05.377 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.395+0000 7fe94159a640 1 -- 172.21.15.17:0/2998963914 <== mon.2 v1:172.21.15.17:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fe938001c00 con 0x7fe944142690 2024-03-29T11:20:05.377 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.396+0000 7fe94159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:05.377 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.396+0000 7fe94159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:05.378 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.396+0000 7fe94159a640 10 monclient: dump: 2024-03-29T11:20:05.378 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:20:05.378 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:20:05.378 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:05.378 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:05.378 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:05.378 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:05.378 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:20:05.378 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:20:05.378 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:20:05.378 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:05.378 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.396+0000 7fe9489a4640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:05.378 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.396+0000 7fe9489a4640 4 set_mon_vals no callback set 2024-03-29T11:20:05.378 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.397+0000 7fe9489a4640 10 monclient: shutdown 2024-03-29T11:20:05.379 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.397+0000 7fe9489a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:05.379 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.397+0000 7fe9489a4640 1 -- 172.21.15.17:0/2998963914 >> v1:172.21.15.17:6790/0 conn(0x7fe944142690 legacy=0x7fe9440b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:05.379 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.397+0000 7fe9489a4640 1 -- 172.21.15.17:0/2998963914 shutdown_connections 2024-03-29T11:20:05.379 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.397+0000 7fe9489a4640 1 -- 172.21.15.17:0/2998963914 >> 172.21.15.17:0/2998963914 conn(0x7fe944066a00 msgr2=0x7fe944066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:05.379 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.397+0000 7fe9489a4640 1 -- 172.21.15.17:0/2998963914 shutdown_connections 2024-03-29T11:20:05.379 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.397+0000 7fe9489a4640 1 -- 172.21.15.17:0/2998963914 wait complete. 2024-03-29T11:20:05.382 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.399+0000 7fe9489a4640 10 monclient: build_initial_monmap 2024-03-29T11:20:05.382 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.399+0000 7fe9489a4640 10 monclient: monmap: 2024-03-29T11:20:05.382 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:20:05.383 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:05.383 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:20:05.383 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:20:05.383 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:05.383 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:05.383 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:20:05.383 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:20:05.383 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:20:05.383 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:05.383 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.399+0000 7fe9489a4640 1 librados: starting msgr at 2024-03-29T11:20:05.383 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.399+0000 7fe9489a4640 1 librados: starting objecter 2024-03-29T11:20:05.383 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.399+0000 7fe9489a4640 1 Processor -- start 2024-03-29T11:20:05.383 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.399+0000 7fe9489a4640 1 -- start start 2024-03-29T11:20:05.383 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.399+0000 7fe9489a4640 1 librados: setting wanted keys 2024-03-29T11:20:05.384 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.399+0000 7fe9489a4640 1 librados: calling monclient init 2024-03-29T11:20:05.384 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.399+0000 7fe9489a4640 10 monclient: init 2024-03-29T11:20:05.384 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.400+0000 7fe9489a4640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:05.384 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.400+0000 7fe9489a4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:20:05.384 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.400+0000 7fe9489a4640 10 monclient(hunting): picked mon.noname-b con 0x7fe9440b17d0 addr v1:172.21.15.94:6789/0 2024-03-29T11:20:05.384 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.400+0000 7fe9489a4640 10 monclient(hunting): picked mon.noname-a con 0x7fe9441422b0 addr v1:172.21.15.17:6789/0 2024-03-29T11:20:05.384 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.400+0000 7fe9489a4640 10 monclient(hunting): picked mon.noname-c con 0x7fe944142690 addr v1:172.21.15.17:6790/0 2024-03-29T11:20:05.384 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.400+0000 7fe9489a4640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9440ad0b0 con 0x7fe9441422b0 2024-03-29T11:20:05.384 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.400+0000 7fe9489a4640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9440abf50 con 0x7fe9440b17d0 2024-03-29T11:20:05.384 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.400+0000 7fe9489a4640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe944132620 con 0x7fe944142690 2024-03-29T11:20:05.384 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.401+0000 7fe9489a4640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:05.384 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.401+0000 7fe9489a4640 10 monclient(hunting): authenticate will time out at 911.605894s 2024-03-29T11:20:05.384 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.401+0000 7fe941d9b640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fe9441422b0 0x7fe9440b09a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:39002/0 (socket says 172.21.15.17:39002) 2024-03-29T11:20:05.384 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.401+0000 7fe941d9b640 1 -- 172.21.15.17:0/2052169708 learned_addr learned my addr 172.21.15.17:0/2052169708 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:20:05.385 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.401+0000 7fe932ffd640 10 client.?.objecter ms_handle_connect 0x7fe9441422b0 2024-03-29T11:20:05.385 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.401+0000 7fe932ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:05.385 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.401+0000 7fe932ffd640 10 client.?.objecter ms_handle_connect 0x7fe944142690 2024-03-29T11:20:05.385 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.401+0000 7fe932ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:05.385 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.401+0000 7fe932ffd640 10 client.?.objecter ms_handle_connect 0x7fe9440b17d0 2024-03-29T11:20:05.385 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.401+0000 7fe932ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:05.385 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.401+0000 7fe932ffd640 1 -- 172.21.15.17:0/2052169708 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 677647920 0 0) 0x7fe9440ad0b0 con 0x7fe9441422b0 2024-03-29T11:20:05.385 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.401+0000 7fe932ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:05.385 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.401+0000 7fe932ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:05.385 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.401+0000 7fe932ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:05.385 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.401+0000 7fe932ffd640 10 monclient(hunting): my global_id is 4263 2024-03-29T11:20:05.385 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.401+0000 7fe932ffd640 1 -- 172.21.15.17:0/2052169708 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe924004eb0 con 0x7fe9441422b0 2024-03-29T11:20:05.385 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.401+0000 7fe932ffd640 1 -- 172.21.15.17:0/2052169708 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 145663088 0 0) 0x7fe9440abf50 con 0x7fe9440b17d0 2024-03-29T11:20:05.385 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.401+0000 7fe932ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:05.385 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.402+0000 7fe932ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:05.386 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.402+0000 7fe932ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:05.386 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.402+0000 7fe932ffd640 10 monclient(hunting): my global_id is 4279 2024-03-29T11:20:05.386 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.402+0000 7fe932ffd640 1 -- 172.21.15.17:0/2052169708 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9440ad0b0 con 0x7fe9440b17d0 2024-03-29T11:20:05.386 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.402+0000 7fe932ffd640 1 -- 172.21.15.17:0/2052169708 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1347283146 0 0) 0x7fe944132620 con 0x7fe944142690 2024-03-29T11:20:05.386 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.402+0000 7fe932ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:05.386 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.402+0000 7fe932ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:05.386 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.402+0000 7fe932ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:05.386 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.402+0000 7fe932ffd640 10 monclient(hunting): my global_id is 4271 2024-03-29T11:20:05.386 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.402+0000 7fe932ffd640 1 -- 172.21.15.17:0/2052169708 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9440abf50 con 0x7fe944142690 2024-03-29T11:20:05.386 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.402+0000 7fe932ffd640 1 -- 172.21.15.17:0/2052169708 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1684549094 0 0) 0x7fe924004eb0 con 0x7fe9441422b0 2024-03-29T11:20:05.386 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.402+0000 7fe932ffd640 1 -- 172.21.15.17:0/2052169708 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe944132620 con 0x7fe9441422b0 2024-03-29T11:20:05.386 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.402+0000 7fe932ffd640 1 -- 172.21.15.17:0/2052169708 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fe938001c80 con 0x7fe9441422b0 2024-03-29T11:20:05.386 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.402+0000 7fe932ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:05.387 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.402+0000 7fe932ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:05.387 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.402+0000 7fe932ffd640 10 monclient(hunting): dump: 2024-03-29T11:20:05.387 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:20:05.387 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:20:05.387 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:05.387 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:05.387 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:05.387 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:05.387 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:20:05.387 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:20:05.387 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:20:05.387 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:05.387 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.402+0000 7fe932ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:05.387 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.402+0000 7fe932ffd640 1 -- 172.21.15.17:0/2052169708 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 300188636 0 0) 0x7fe9440ad0b0 con 0x7fe9440b17d0 2024-03-29T11:20:05.387 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.402+0000 7fe932ffd640 1 -- 172.21.15.17:0/2052169708 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe924004eb0 con 0x7fe9440b17d0 2024-03-29T11:20:05.387 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.402+0000 7fe932ffd640 1 -- 172.21.15.17:0/2052169708 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fe934001820 con 0x7fe9440b17d0 2024-03-29T11:20:05.387 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.402+0000 7fe932ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:05.388 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.402+0000 7fe932ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:05.388 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.402+0000 7fe932ffd640 10 monclient(hunting): dump: 2024-03-29T11:20:05.388 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:20:05.388 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:20:05.388 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:05.388 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:05.388 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:05.388 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:05.388 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:20:05.388 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:20:05.388 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:20:05.388 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:05.388 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.402+0000 7fe932ffd640 1 -- 172.21.15.17:0/2052169708 <== mon.0 v1:172.21.15.17:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 524333062 0 0) 0x7fe944132620 con 0x7fe9441422b0 2024-03-29T11:20:05.388 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.402+0000 7fe932ffd640 1 -- 172.21.15.17:0/2052169708 >> v1:172.21.15.17:6790/0 conn(0x7fe944142690 legacy=0x7fe944130060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:05.388 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.402+0000 7fe932ffd640 1 -- 172.21.15.17:0/2052169708 >> v1:172.21.15.94:6789/0 conn(0x7fe9440b17d0 legacy=0x7fe944131f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:05.388 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.402+0000 7fe932ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:05.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.402+0000 7fe932ffd640 1 monclient: found mon.a 2024-03-29T11:20:05.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.402+0000 7fe932ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:05.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.402+0000 7fe932ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:20:05.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.402+0000 7fe932ffd640 1 -- 172.21.15.17:0/2052169708 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9441327b0 con 0x7fe9441422b0 2024-03-29T11:20:05.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.402+0000 7fe932ffd640 10 monclient: _finish_auth 0 2024-03-29T11:20:05.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.402+0000 7fe932ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:05.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.402+0000 7fe932ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:05.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.403+0000 7fe9489a4640 5 monclient: authenticate success, global_id 4263 2024-03-29T11:20:05.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.403+0000 7fe9489a4640 10 monclient: _renew_subs 2024-03-29T11:20:05.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.403+0000 7fe9489a4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:20:05.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.403+0000 7fe9489a4640 1 -- 172.21.15.17:0/2052169708 --> v1:172.21.15.17:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9440b10b0 con 0x7fe9441422b0 2024-03-29T11:20:05.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.403+0000 7fe9489a4640 10 client.4263.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:05.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.403+0000 7fe9489a4640 10 monclient: _renew_subs 2024-03-29T11:20:05.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.403+0000 7fe9489a4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:20:05.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.403+0000 7fe9489a4640 1 -- 172.21.15.17:0/2052169708 --> v1:172.21.15.17:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9440abb30 con 0x7fe9441422b0 2024-03-29T11:20:05.390 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.403+0000 7fe9489a4640 1 librados: init done 2024-03-29T11:20:05.390 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.403+0000 7fe932ffd640 1 -- 172.21.15.17:0/2052169708 <== mon.0 v1:172.21.15.17:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe938004a90 con 0x7fe9441422b0 2024-03-29T11:20:05.390 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.403+0000 7fe932ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:05.390 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.403+0000 7fe932ffd640 1 -- 172.21.15.17:0/2052169708 <== mon.0 v1:172.21.15.17:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fe938004da0 con 0x7fe9441422b0 2024-03-29T11:20:05.390 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.403+0000 7fe932ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:05.390 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.403+0000 7fe932ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:05.390 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.403+0000 7fe932ffd640 10 monclient: dump: 2024-03-29T11:20:05.390 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:20:05.390 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:20:05.390 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:05.390 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:05.390 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:05.390 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:05.390 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:20:05.391 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:20:05.391 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:20:05.391 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:05.391 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.403+0000 7fe940d99640 4 set_mon_vals no callback set 2024-03-29T11:20:05.391 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.404+0000 7fe9489a4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:05.391 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.405+0000 7fe9489a4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:05.391 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.405+0000 7fe9489a4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:20:05.391 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.405+0000 7fe9489a4640 1 -- 172.21.15.17:0/2052169708 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe928005080 con 0x7fe9441422b0 2024-03-29T11:20:05.391 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.406+0000 7fe932ffd640 1 -- 172.21.15.17:0/2052169708 <== mon.0 v1:172.21.15.17:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7fe93801ad70 con 0x7fe9441422b0 2024-03-29T11:20:05.392 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.407+0000 7fe932ffd640 1 -- 172.21.15.17:0/2052169708 <== mon.0 v1:172.21.15.17:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7fe938084bb0 con 0x7fe9441422b0 2024-03-29T11:20:05.392 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.407+0000 7fe932ffd640 10 client.4263.objecter ms_dispatch 0x7fe944000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:05.392 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.407+0000 7fe932ffd640 3 client.4263.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:05.392 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.407+0000 7fe932ffd640 3 client.4263.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:05.392 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.407+0000 7fe932ffd640 20 client.4263.objecter dump_active .. 0 homeless 2024-03-29T11:20:05.392 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.407+0000 7fe94259c640 10 monclient: get_auth_request con 0x7fe92406df50 auth_method 0 2024-03-29T11:20:05.393 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.408+0000 7fe932ffd640 10 client.4263.objecter ms_handle_connect 0x7fe92406df50 2024-03-29T11:20:05.393 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.413+0000 7fe932ffd640 1 -- 172.21.15.17:0/2052169708 <== mon.0 v1:172.21.15.17:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe9380504f0 con 0x7fe9441422b0 2024-03-29T11:20:05.393 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.413+0000 7fe932ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:05.393 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.413+0000 7fe932ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:05.787 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.810+0000 7fe9489a4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:05.787 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.810+0000 7fe9489a4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:05.788 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.810+0000 7fe9489a4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:20:05.788 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.810+0000 7fe9489a4640 1 -- 172.21.15.17:0/2052169708 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe928005330 con 0x7fe9441422b0 2024-03-29T11:20:05.789 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.812+0000 7fe932ffd640 1 -- 172.21.15.17:0/2052169708 <== mon.0 v1:172.21.15.17:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7fe938052420 con 0x7fe9441422b0 2024-03-29T11:20:05.789 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.812+0000 7fe932ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:05.789 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.812+0000 7fe932ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:05.789 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:20:05.789 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:05.793 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.816+0000 7fe9489a4640 10 librados: watch_flush enter 2024-03-29T11:20:05.794 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.816+0000 7fe9489a4640 10 librados: watch_flush exit 2024-03-29T11:20:05.794 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.816+0000 7fe9489a4640 20 client.4263.objecter shutdown clearing up homeless session... 2024-03-29T11:20:05.794 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.816+0000 7fe9489a4640 10 client.4263.objecter successfully canceled tick 2024-03-29T11:20:05.794 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.816+0000 7fe9489a4640 1 -- 172.21.15.17:0/2052169708 >> v1:172.21.15.17:6816/3236185052 conn(0x7fe92406df50 legacy=0x7fe924070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:05.794 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.816+0000 7fe9489a4640 10 monclient: shutdown 2024-03-29T11:20:05.794 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.816+0000 7fe9489a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:05.794 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.816+0000 7fe9489a4640 1 -- 172.21.15.17:0/2052169708 >> v1:172.21.15.17:6789/0 conn(0x7fe9441422b0 legacy=0x7fe9440b09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:05.794 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.816+0000 7fe9489a4640 1 -- 172.21.15.17:0/2052169708 shutdown_connections 2024-03-29T11:20:05.794 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.816+0000 7fe9489a4640 1 -- 172.21.15.17:0/2052169708 >> 172.21.15.17:0/2052169708 conn(0x7fe944066a00 msgr2=0x7fe94413dfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:05.794 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.817+0000 7fe9489a4640 1 -- 172.21.15.17:0/2052169708 shutdown_connections 2024-03-29T11:20:05.794 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.817+0000 7fe9489a4640 1 -- 172.21.15.17:0/2052169708 wait complete. 2024-03-29T11:20:05.794 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:05.817+0000 7fe9489a4640 1 librados: shutdown 2024-03-29T11:20:05.809 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:12.615 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:12.767 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.790+0000 7f86bf8bb640 10 monclient: get_monmap_and_config 2024-03-29T11:20:12.767 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.790+0000 7f86bf8bb640 10 monclient: build_initial_monmap 2024-03-29T11:20:12.767 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.790+0000 7f86bf8bb640 10 monclient: monmap: 2024-03-29T11:20:12.767 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:20:12.768 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:20:12.768 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:20:12.791774+0000 2024-03-29T11:20:12.768 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:20:12.791774+0000 2024-03-29T11:20:12.768 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:12.768 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:12.768 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:20:12.768 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:20:12.768 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:20:12.768 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:12.773 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.791+0000 7f86bf8bb640 1 Processor -- start 2024-03-29T11:20:12.773 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.792+0000 7f86bf8bb640 1 -- start start 2024-03-29T11:20:12.773 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.792+0000 7f86bf8bb640 10 monclient: init 2024-03-29T11:20:12.773 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.792+0000 7f86bf8bb640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:12.773 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.792+0000 7f86bf8bb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:12.773 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.792+0000 7f86bf8bb640 10 monclient(hunting): picked mon.noname-a con 0x7f86b81422b0 addr v1:172.21.15.17:6789/0 2024-03-29T11:20:12.773 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.792+0000 7f86bf8bb640 10 monclient(hunting): picked mon.noname-c con 0x7f86b8142690 addr v1:172.21.15.94:6789/0 2024-03-29T11:20:12.773 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.792+0000 7f86bf8bb640 10 monclient(hunting): picked mon.noname-b con 0x7f86b80b17d0 addr v1:172.21.15.17:6790/0 2024-03-29T11:20:12.773 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.792+0000 7f86bf8bb640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86b8142c80 con 0x7f86b81422b0 2024-03-29T11:20:12.773 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.792+0000 7f86bf8bb640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86b80aae40 con 0x7f86b8142690 2024-03-29T11:20:12.773 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.792+0000 7f86bf8bb640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86b813fbd0 con 0x7f86b80b17d0 2024-03-29T11:20:12.773 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.792+0000 7f86bf8bb640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:12.774 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.792+0000 7f86bf8bb640 10 monclient(hunting): authenticate will time out at 918.997752s 2024-03-29T11:20:12.774 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.793+0000 7f86bce56640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f86b8142690 0x7f86b80b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:35052/0 (socket says 172.21.15.17:35052) 2024-03-29T11:20:12.774 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.793+0000 7f86bce56640 1 -- 172.21.15.17:0/3189184500 learned_addr learned my addr 172.21.15.17:0/3189184500 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:20:12.774 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.793+0000 7f86affff640 1 -- 172.21.15.17:0/3189184500 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 51677563 0 0) 0x7f86b813fbd0 con 0x7f86b80b17d0 2024-03-29T11:20:12.774 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.793+0000 7f86affff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:12.774 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.793+0000 7f86affff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:12.774 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.793+0000 7f86affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:12.774 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.793+0000 7f86affff640 10 monclient(hunting): my global_id is 4274 2024-03-29T11:20:12.774 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.793+0000 7f86affff640 1 -- 172.21.15.17:0/3189184500 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86a4003520 con 0x7f86b80b17d0 2024-03-29T11:20:12.774 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.794+0000 7f86affff640 1 -- 172.21.15.17:0/3189184500 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2397712050 0 0) 0x7f86b80aae40 con 0x7f86b8142690 2024-03-29T11:20:12.774 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.794+0000 7f86affff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:12.774 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.794+0000 7f86affff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:12.774 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.794+0000 7f86affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:12.774 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.794+0000 7f86affff640 10 monclient(hunting): my global_id is 4282 2024-03-29T11:20:12.775 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.794+0000 7f86affff640 1 -- 172.21.15.17:0/3189184500 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86a4003840 con 0x7f86b8142690 2024-03-29T11:20:12.775 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.794+0000 7f86affff640 1 -- 172.21.15.17:0/3189184500 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2765808185 0 0) 0x7f86b8142c80 con 0x7f86b81422b0 2024-03-29T11:20:12.775 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.794+0000 7f86affff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:12.775 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.794+0000 7f86affff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:12.775 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.794+0000 7f86affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:12.775 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.794+0000 7f86affff640 10 monclient(hunting): my global_id is 4266 2024-03-29T11:20:12.775 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.794+0000 7f86affff640 1 -- 172.21.15.17:0/3189184500 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86b80aae40 con 0x7f86b81422b0 2024-03-29T11:20:12.775 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.794+0000 7f86affff640 1 -- 172.21.15.17:0/3189184500 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 470036891 0 0) 0x7f86a4003840 con 0x7f86b8142690 2024-03-29T11:20:12.775 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.794+0000 7f86affff640 1 -- 172.21.15.17:0/3189184500 >> v1:172.21.15.17:6790/0 conn(0x7f86b80b17d0 legacy=0x7f86b80b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:12.775 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.794+0000 7f86affff640 1 -- 172.21.15.17:0/3189184500 >> v1:172.21.15.17:6789/0 conn(0x7f86b81422b0 legacy=0x7f86b813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:12.775 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.794+0000 7f86affff640 10 monclient: _finish_hunting 0 2024-03-29T11:20:12.775 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.794+0000 7f86affff640 1 monclient: found mon.noname-c 2024-03-29T11:20:12.775 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.794+0000 7f86affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:12.776 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.794+0000 7f86affff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6789/0 2024-03-29T11:20:12.776 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.794+0000 7f86affff640 1 -- 172.21.15.17:0/3189184500 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86b8151940 con 0x7f86b8142690 2024-03-29T11:20:12.776 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.794+0000 7f86affff640 10 monclient: _finish_auth 0 2024-03-29T11:20:12.776 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.794+0000 7f86affff640 10 monclient: _check_auth_tickets 2024-03-29T11:20:12.776 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.794+0000 7f86affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:12.776 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.794+0000 7f86affff640 1 -- 172.21.15.17:0/3189184500 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f86a8001670 con 0x7f86b8142690 2024-03-29T11:20:12.776 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.794+0000 7f86affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:12.776 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.794+0000 7f86affff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:12.776 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.794+0000 7f86affff640 10 monclient: dump: 2024-03-29T11:20:12.776 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:20:12.776 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:20:12.776 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:12.776 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:12.776 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:12.777 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:12.777 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:20:12.777 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:20:12.777 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:20:12.777 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:12.777 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.794+0000 7f86bf8bb640 5 monclient: authenticate success, global_id 4282 2024-03-29T11:20:12.777 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.794+0000 7f86bf8bb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:12.777 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.795+0000 7f86affff640 1 -- 172.21.15.17:0/3189184500 <== mon.1 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86a80017d0 con 0x7f86b8142690 2024-03-29T11:20:12.777 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.795+0000 7f86affff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:12.777 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.795+0000 7f86affff640 1 -- 172.21.15.17:0/3189184500 <== mon.1 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f86a8001b60 con 0x7f86b8142690 2024-03-29T11:20:12.777 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.795+0000 7f86affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:12.777 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.795+0000 7f86affff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:12.777 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.795+0000 7f86affff640 10 monclient: dump: 2024-03-29T11:20:12.777 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:20:12.778 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:20:12.778 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:12.778 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:12.778 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:12.778 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:12.778 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:20:12.778 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:20:12.778 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:20:12.778 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:12.778 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.795+0000 7f86bf8bb640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:12.778 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.795+0000 7f86bf8bb640 4 set_mon_vals no callback set 2024-03-29T11:20:12.778 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.796+0000 7f86bf8bb640 10 monclient: shutdown 2024-03-29T11:20:12.778 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.796+0000 7f86bf8bb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:12.778 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.796+0000 7f86bf8bb640 1 -- 172.21.15.17:0/3189184500 >> v1:172.21.15.94:6789/0 conn(0x7f86b8142690 legacy=0x7f86b80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:12.779 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.796+0000 7f86bf8bb640 1 -- 172.21.15.17:0/3189184500 shutdown_connections 2024-03-29T11:20:12.779 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.796+0000 7f86bf8bb640 1 -- 172.21.15.17:0/3189184500 >> 172.21.15.17:0/3189184500 conn(0x7f86b8066a00 msgr2=0x7f86b8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:12.779 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.796+0000 7f86bf8bb640 1 -- 172.21.15.17:0/3189184500 shutdown_connections 2024-03-29T11:20:12.781 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.796+0000 7f86bf8bb640 1 -- 172.21.15.17:0/3189184500 wait complete. 2024-03-29T11:20:12.782 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.797+0000 7f86bf8bb640 10 monclient: build_initial_monmap 2024-03-29T11:20:12.782 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.797+0000 7f86bf8bb640 10 monclient: monmap: 2024-03-29T11:20:12.782 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:20:12.782 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:12.782 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:20:12.782 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:20:12.782 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:12.782 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:12.782 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:20:12.782 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:20:12.782 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:20:12.782 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:12.783 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.798+0000 7f86bf8bb640 1 librados: starting msgr at 2024-03-29T11:20:12.783 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.798+0000 7f86bf8bb640 1 librados: starting objecter 2024-03-29T11:20:12.783 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.798+0000 7f86bf8bb640 1 Processor -- start 2024-03-29T11:20:12.783 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.798+0000 7f86bf8bb640 1 -- start start 2024-03-29T11:20:12.783 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.798+0000 7f86bf8bb640 1 librados: setting wanted keys 2024-03-29T11:20:12.783 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.798+0000 7f86bf8bb640 1 librados: calling monclient init 2024-03-29T11:20:12.783 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.798+0000 7f86bf8bb640 10 monclient: init 2024-03-29T11:20:12.783 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.799+0000 7f86bf8bb640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:12.783 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.799+0000 7f86bf8bb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:20:12.783 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.799+0000 7f86bf8bb640 10 monclient(hunting): picked mon.noname-b con 0x7f86b80b17d0 addr v1:172.21.15.94:6789/0 2024-03-29T11:20:12.783 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.799+0000 7f86bf8bb640 10 monclient(hunting): picked mon.noname-c con 0x7f86b81422b0 addr v1:172.21.15.17:6790/0 2024-03-29T11:20:12.783 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.799+0000 7f86bd657640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f86b80b17d0 0x7f86b8071a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:40838/0 (socket says 172.21.15.17:40838) 2024-03-29T11:20:12.783 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.799+0000 7f86bf8bb640 10 monclient(hunting): picked mon.noname-a con 0x7f86b8142690 addr v1:172.21.15.17:6789/0 2024-03-29T11:20:12.783 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.799+0000 7f86bd657640 1 -- 172.21.15.17:0/2326651298 learned_addr learned my addr 172.21.15.17:0/2326651298 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:20:12.784 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.799+0000 7f86bf8bb640 1 -- 172.21.15.17:0/2326651298 --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86b8070cd0 con 0x7f86b8142690 2024-03-29T11:20:12.784 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.799+0000 7f86bf8bb640 1 -- 172.21.15.17:0/2326651298 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86b8070fb0 con 0x7f86b80b17d0 2024-03-29T11:20:12.784 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.799+0000 7f86bf8bb640 1 -- 172.21.15.17:0/2326651298 --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86b81db9d0 con 0x7f86b81422b0 2024-03-29T11:20:12.784 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.799+0000 7f86bf8bb640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:12.784 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.799+0000 7f86bf8bb640 10 monclient(hunting): authenticate will time out at 919.004844s 2024-03-29T11:20:12.784 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.800+0000 7f86adffb640 10 client.?.objecter ms_handle_connect 0x7f86b80b17d0 2024-03-29T11:20:12.784 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.800+0000 7f86adffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:12.784 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.800+0000 7f86adffb640 1 -- 172.21.15.17:0/2326651298 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4183267040 0 0) 0x7f86b8070fb0 con 0x7f86b80b17d0 2024-03-29T11:20:12.784 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.800+0000 7f86adffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:12.784 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.800+0000 7f86adffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:12.784 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.800+0000 7f86adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:12.784 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.800+0000 7f86adffb640 10 monclient(hunting): my global_id is 4285 2024-03-29T11:20:12.784 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.800+0000 7f86adffb640 1 -- 172.21.15.17:0/2326651298 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f869c004eb0 con 0x7f86b80b17d0 2024-03-29T11:20:12.784 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.800+0000 7f86adffb640 10 client.?.objecter ms_handle_connect 0x7f86b81422b0 2024-03-29T11:20:12.784 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.800+0000 7f86adffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:12.785 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.800+0000 7f86adffb640 1 -- 172.21.15.17:0/2326651298 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1145144371 0 0) 0x7f86b81db9d0 con 0x7f86b81422b0 2024-03-29T11:20:12.785 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.800+0000 7f86adffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:12.785 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.800+0000 7f86adffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:12.785 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.800+0000 7f86adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:12.785 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.800+0000 7f86adffb640 10 monclient(hunting): my global_id is 4277 2024-03-29T11:20:12.785 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.800+0000 7f86adffb640 1 -- 172.21.15.17:0/2326651298 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86b8070fb0 con 0x7f86b81422b0 2024-03-29T11:20:12.785 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.800+0000 7f86adffb640 10 client.?.objecter ms_handle_connect 0x7f86b8142690 2024-03-29T11:20:12.785 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.800+0000 7f86adffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:12.785 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.800+0000 7f86adffb640 1 -- 172.21.15.17:0/2326651298 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2098892079 0 0) 0x7f869c004eb0 con 0x7f86b80b17d0 2024-03-29T11:20:12.785 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.801+0000 7f86adffb640 1 -- 172.21.15.17:0/2326651298 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86b81db9d0 con 0x7f86b80b17d0 2024-03-29T11:20:12.785 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.801+0000 7f86adffb640 1 -- 172.21.15.17:0/2326651298 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f86b4001a30 con 0x7f86b80b17d0 2024-03-29T11:20:12.785 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.801+0000 7f86adffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:12.785 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.801+0000 7f86adffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:12.785 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.801+0000 7f86adffb640 10 monclient(hunting): dump: 2024-03-29T11:20:12.786 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:20:12.786 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:20:12.786 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:12.786 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:12.786 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:12.786 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:12.786 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:20:12.786 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:20:12.786 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:20:12.786 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:12.786 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.801+0000 7f86adffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:12.786 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.801+0000 7f86adffb640 1 -- 172.21.15.17:0/2326651298 <== mon.1 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2466541161 0 0) 0x7f86b81db9d0 con 0x7f86b80b17d0 2024-03-29T11:20:12.786 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.801+0000 7f86adffb640 1 -- 172.21.15.17:0/2326651298 >> v1:172.21.15.17:6790/0 conn(0x7f86b81422b0 legacy=0x7f86b81dbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:12.786 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.801+0000 7f86adffb640 1 -- 172.21.15.17:0/2326651298 >> v1:172.21.15.17:6789/0 conn(0x7f86b8142690 legacy=0x7f86b813bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:12.786 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.801+0000 7f86adffb640 10 monclient: _finish_hunting 0 2024-03-29T11:20:12.787 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.801+0000 7f86adffb640 1 monclient: found mon.b 2024-03-29T11:20:12.787 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.801+0000 7f86adffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:12.787 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.801+0000 7f86adffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:20:12.787 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.801+0000 7f86adffb640 1 -- 172.21.15.17:0/2326651298 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86b80b1020 con 0x7f86b80b17d0 2024-03-29T11:20:12.787 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.801+0000 7f86adffb640 10 monclient: _finish_auth 0 2024-03-29T11:20:12.787 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.801+0000 7f86adffb640 10 monclient: _check_auth_tickets 2024-03-29T11:20:12.787 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.801+0000 7f86adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:12.787 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.801+0000 7f86adffb640 1 -- 172.21.15.17:0/2326651298 <== mon.1 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86b4001c70 con 0x7f86b80b17d0 2024-03-29T11:20:12.787 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.801+0000 7f86adffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:12.787 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.801+0000 7f86adffb640 1 -- 172.21.15.17:0/2326651298 <== mon.1 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f86b4003a60 con 0x7f86b80b17d0 2024-03-29T11:20:12.787 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.801+0000 7f86bf8bb640 5 monclient: authenticate success, global_id 4285 2024-03-29T11:20:12.787 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.801+0000 7f86bf8bb640 10 monclient: _renew_subs 2024-03-29T11:20:12.787 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.801+0000 7f86bf8bb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:20:12.788 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.801+0000 7f86bf8bb640 1 -- 172.21.15.17:0/2326651298 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86b81dc600 con 0x7f86b80b17d0 2024-03-29T11:20:12.788 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.802+0000 7f86adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:12.788 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.802+0000 7f86adffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:12.788 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.802+0000 7f86adffb640 10 monclient: dump: 2024-03-29T11:20:12.788 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:20:12.788 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:20:12.788 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:12.788 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:12.788 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:12.788 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:12.788 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:20:12.788 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:20:12.789 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:20:12.789 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:12.789 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.801+0000 7f86af7fe640 4 set_mon_vals no callback set 2024-03-29T11:20:12.789 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.802+0000 7f86bf8bb640 10 client.4285.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:12.789 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.802+0000 7f86bf8bb640 10 monclient: _renew_subs 2024-03-29T11:20:12.789 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.802+0000 7f86bf8bb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:20:12.789 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.802+0000 7f86bf8bb640 1 -- 172.21.15.17:0/2326651298 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f86b806ee10 con 0x7f86b80b17d0 2024-03-29T11:20:12.789 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.802+0000 7f86bf8bb640 1 librados: init done 2024-03-29T11:20:12.789 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.804+0000 7f86adffb640 1 -- 172.21.15.17:0/2326651298 <== mon.1 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f86b4003bc0 con 0x7f86b80b17d0 2024-03-29T11:20:12.789 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.804+0000 7f86bf8bb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:12.789 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.804+0000 7f86bf8bb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:12.789 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.804+0000 7f86bf8bb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:20:12.789 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.804+0000 7f86bf8bb640 1 -- 172.21.15.17:0/2326651298 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86b81dbb60 con 0x7f86b80b17d0 2024-03-29T11:20:12.789 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.805+0000 7f86adffb640 1 -- 172.21.15.17:0/2326651298 <== mon.1 v1:172.21.15.94:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f86b4084ac0 con 0x7f86b80b17d0 2024-03-29T11:20:12.790 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.805+0000 7f86adffb640 10 client.4285.objecter ms_dispatch 0x7f86b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:12.790 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.805+0000 7f86adffb640 3 client.4285.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:12.790 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.805+0000 7f86adffb640 3 client.4285.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:12.790 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.805+0000 7f86adffb640 20 client.4285.objecter dump_active .. 0 homeless 2024-03-29T11:20:12.790 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.805+0000 7f86bce56640 10 monclient: get_auth_request con 0x7f869c06dd60 auth_method 0 2024-03-29T11:20:12.790 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.805+0000 7f86adffb640 10 client.4285.objecter ms_handle_connect 0x7f869c06dd60 2024-03-29T11:20:12.790 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.812+0000 7f86adffb640 1 -- 172.21.15.17:0/2326651298 <== mon.1 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f86b40503a0 con 0x7f86b80b17d0 2024-03-29T11:20:12.790 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.812+0000 7f86adffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:12.791 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:12.812+0000 7f86adffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:13.183 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:13.205+0000 7f86bf8bb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:13.184 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:13.205+0000 7f86bf8bb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:13.184 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:13.205+0000 7f86bf8bb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:20:13.184 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:13.205+0000 7f86bf8bb640 1 -- 172.21.15.17:0/2326651298 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f86b8075f10 con 0x7f86b80b17d0 2024-03-29T11:20:13.184 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:13.206+0000 7f86adffb640 1 -- 172.21.15.17:0/2326651298 <== mon.1 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f86b40522d0 con 0x7f86b80b17d0 2024-03-29T11:20:13.184 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:13.206+0000 7f86adffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:13.184 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:13.206+0000 7f86adffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:13.185 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:20:13.185 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:13.186 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:13.210+0000 7f86bf8bb640 10 librados: watch_flush enter 2024-03-29T11:20:13.188 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:13.211+0000 7f86bf8bb640 10 librados: watch_flush exit 2024-03-29T11:20:13.188 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:13.211+0000 7f86bf8bb640 20 client.4285.objecter shutdown clearing up homeless session... 2024-03-29T11:20:13.189 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:13.211+0000 7f86bf8bb640 10 client.4285.objecter successfully canceled tick 2024-03-29T11:20:13.189 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:13.211+0000 7f86bf8bb640 1 -- 172.21.15.17:0/2326651298 >> v1:172.21.15.17:6816/3236185052 conn(0x7f869c06dd60 legacy=0x7f869c070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:13.189 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:13.211+0000 7f86bf8bb640 10 monclient: shutdown 2024-03-29T11:20:13.189 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:13.211+0000 7f86bf8bb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:13.189 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:13.211+0000 7f86bf8bb640 1 -- 172.21.15.17:0/2326651298 >> v1:172.21.15.94:6789/0 conn(0x7f86b80b17d0 legacy=0x7f86b8071a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:13.189 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:13.212+0000 7f86bf8bb640 1 -- 172.21.15.17:0/2326651298 shutdown_connections 2024-03-29T11:20:13.189 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:13.212+0000 7f86bf8bb640 1 -- 172.21.15.17:0/2326651298 >> 172.21.15.17:0/2326651298 conn(0x7f86b8066a00 msgr2=0x7f86b80b3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:13.189 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:13.212+0000 7f86bf8bb640 1 -- 172.21.15.17:0/2326651298 shutdown_connections 2024-03-29T11:20:13.189 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:13.212+0000 7f86bf8bb640 1 -- 172.21.15.17:0/2326651298 wait complete. 2024-03-29T11:20:13.189 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:13.212+0000 7f86bf8bb640 1 librados: shutdown 2024-03-29T11:20:13.205 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:20.013 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:20.152 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.174+0000 7f92f876b640 10 monclient: get_monmap_and_config 2024-03-29T11:20:20.152 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.174+0000 7f92f876b640 10 monclient: build_initial_monmap 2024-03-29T11:20:20.152 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.174+0000 7f92f876b640 10 monclient: monmap: 2024-03-29T11:20:20.152 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:20:20.152 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:20:20.152 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:20:20.175880+0000 2024-03-29T11:20:20.152 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:20:20.175880+0000 2024-03-29T11:20:20.152 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:20.152 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:20.152 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:20:20.152 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:20:20.152 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:20:20.153 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:20.153 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.175+0000 7f92f876b640 1 Processor -- start 2024-03-29T11:20:20.153 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.175+0000 7f92f876b640 1 -- start start 2024-03-29T11:20:20.153 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.175+0000 7f92f876b640 10 monclient: init 2024-03-29T11:20:20.154 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.176+0000 7f92f876b640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:20.154 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.176+0000 7f92f876b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:20.154 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.176+0000 7f92f876b640 10 monclient(hunting): picked mon.noname-a con 0x7f92f01457f0 addr v1:172.21.15.17:6789/0 2024-03-29T11:20:20.154 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.176+0000 7f92f876b640 10 monclient(hunting): picked mon.noname-c con 0x7f92f013f650 addr v1:172.21.15.94:6789/0 2024-03-29T11:20:20.154 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.177+0000 7f92f5d06640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f92f013f650 0x7f92f00b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:57462/0 (socket says 172.21.15.17:57462) 2024-03-29T11:20:20.154 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.177+0000 7f92f6d08640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f92f00a9d70 0x7f92f014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:36498/0 (socket says 172.21.15.17:36498) 2024-03-29T11:20:20.154 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.177+0000 7f92f6d08640 1 -- 172.21.15.17:0/3617231030 learned_addr learned my addr 172.21.15.17:0/3617231030 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:20:20.154 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.177+0000 7f92f876b640 10 monclient(hunting): picked mon.noname-b con 0x7f92f00a9d70 addr v1:172.21.15.17:6790/0 2024-03-29T11:20:20.154 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.177+0000 7f92f876b640 1 -- 172.21.15.17:0/3617231030 --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f00aaef0 con 0x7f92f01457f0 2024-03-29T11:20:20.155 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.177+0000 7f92f876b640 1 -- 172.21.15.17:0/3617231030 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f0145bd0 con 0x7f92f013f650 2024-03-29T11:20:20.155 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.177+0000 7f92f876b640 1 -- 172.21.15.17:0/3617231030 --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f0145eb0 con 0x7f92f00a9d70 2024-03-29T11:20:20.155 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.177+0000 7f92f876b640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:20.155 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.177+0000 7f92f876b640 10 monclient(hunting): authenticate will time out at 926.382364s 2024-03-29T11:20:20.155 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.177+0000 7f92f5505640 1 -- 172.21.15.17:0/3617231030 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1080636009 0 0) 0x7f92f0145bd0 con 0x7f92f013f650 2024-03-29T11:20:20.155 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.177+0000 7f92f5505640 10 monclient(hunting): _negotiate 2024-03-29T11:20:20.155 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.177+0000 7f92f5505640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:20.155 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.177+0000 7f92f5505640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:20.155 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.177+0000 7f92f5505640 10 monclient(hunting): my global_id is 4288 2024-03-29T11:20:20.156 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.177+0000 7f92f5505640 1 -- 172.21.15.17:0/3617231030 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92dc003520 con 0x7f92f013f650 2024-03-29T11:20:20.156 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.177+0000 7f92f5505640 1 -- 172.21.15.17:0/3617231030 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3079557779 0 0) 0x7f92f0145eb0 con 0x7f92f00a9d70 2024-03-29T11:20:20.156 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.178+0000 7f92f5505640 10 monclient(hunting): _negotiate 2024-03-29T11:20:20.156 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.178+0000 7f92f5505640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:20.156 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.178+0000 7f92f5505640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:20.156 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.178+0000 7f92f5505640 10 monclient(hunting): my global_id is 4280 2024-03-29T11:20:20.156 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.178+0000 7f92f5505640 1 -- 172.21.15.17:0/3617231030 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92f0145bd0 con 0x7f92f00a9d70 2024-03-29T11:20:20.156 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.178+0000 7f92f5505640 1 -- 172.21.15.17:0/3617231030 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2712073934 0 0) 0x7f92dc003520 con 0x7f92f013f650 2024-03-29T11:20:20.156 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.178+0000 7f92f5505640 1 -- 172.21.15.17:0/3617231030 >> v1:172.21.15.17:6790/0 conn(0x7f92f00a9d70 legacy=0x7f92f014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:20.156 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.178+0000 7f92f5505640 1 -- 172.21.15.17:0/3617231030 >> v1:172.21.15.17:6789/0 conn(0x7f92f01457f0 legacy=0x7f92f013ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:20.158 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.179+0000 7f92f5505640 10 monclient: _finish_hunting 0 2024-03-29T11:20:20.158 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.179+0000 7f92f5505640 1 monclient: found mon.noname-c 2024-03-29T11:20:20.158 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.179+0000 7f92f5505640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:20.158 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.179+0000 7f92f5505640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6789/0 2024-03-29T11:20:20.158 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.179+0000 7f92f5505640 1 -- 172.21.15.17:0/3617231030 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f0146340 con 0x7f92f013f650 2024-03-29T11:20:20.158 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.179+0000 7f92f5505640 10 monclient: _finish_auth 0 2024-03-29T11:20:20.158 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.179+0000 7f92f5505640 10 monclient: _check_auth_tickets 2024-03-29T11:20:20.158 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.179+0000 7f92f5505640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:20.158 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.179+0000 7f92f5505640 1 -- 172.21.15.17:0/3617231030 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f92e0001690 con 0x7f92f013f650 2024-03-29T11:20:20.158 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.179+0000 7f92f5505640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:20.158 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.179+0000 7f92f5505640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:20.158 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.179+0000 7f92f5505640 10 monclient: dump: 2024-03-29T11:20:20.158 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:20:20.159 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:20:20.159 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:20.159 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:20.159 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:20.159 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:20.159 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:20:20.159 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:20:20.159 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:20:20.159 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:20.159 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.179+0000 7f92f876b640 5 monclient: authenticate success, global_id 4288 2024-03-29T11:20:20.159 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.179+0000 7f92f876b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:20.159 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.179+0000 7f92f5505640 1 -- 172.21.15.17:0/3617231030 <== mon.1 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92e00017f0 con 0x7f92f013f650 2024-03-29T11:20:20.159 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.179+0000 7f92f5505640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:20.159 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.179+0000 7f92f876b640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:20.160 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.179+0000 7f92f876b640 4 set_mon_vals no callback set 2024-03-29T11:20:20.160 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.180+0000 7f92f5505640 1 -- 172.21.15.17:0/3617231030 <== mon.1 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f92e0001bc0 con 0x7f92f013f650 2024-03-29T11:20:20.160 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.180+0000 7f92f5505640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:20.160 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.180+0000 7f92f5505640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:20.160 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.180+0000 7f92f5505640 10 monclient: dump: 2024-03-29T11:20:20.160 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:20:20.160 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:20:20.160 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:20.160 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:20.160 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:20.160 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:20.160 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:20:20.160 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:20:20.161 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:20:20.161 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:20.161 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.181+0000 7f92f876b640 10 monclient: shutdown 2024-03-29T11:20:20.161 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.181+0000 7f92f876b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:20.161 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.181+0000 7f92f876b640 1 -- 172.21.15.17:0/3617231030 >> v1:172.21.15.94:6789/0 conn(0x7f92f013f650 legacy=0x7f92f00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:20.161 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.181+0000 7f92f876b640 1 -- 172.21.15.17:0/3617231030 shutdown_connections 2024-03-29T11:20:20.161 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.181+0000 7f92f876b640 1 -- 172.21.15.17:0/3617231030 >> 172.21.15.17:0/3617231030 conn(0x7f92f0066a00 msgr2=0x7f92f0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:20.161 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.181+0000 7f92f876b640 1 -- 172.21.15.17:0/3617231030 shutdown_connections 2024-03-29T11:20:20.161 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.181+0000 7f92f876b640 1 -- 172.21.15.17:0/3617231030 wait complete. 2024-03-29T11:20:20.164 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.182+0000 7f92f876b640 10 monclient: build_initial_monmap 2024-03-29T11:20:20.164 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.182+0000 7f92f876b640 10 monclient: monmap: 2024-03-29T11:20:20.165 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:20:20.165 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:20.165 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:20:20.165 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:20:20.165 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:20.165 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:20.165 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:20:20.165 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:20:20.165 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:20:20.165 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:20.165 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.182+0000 7f92f876b640 1 librados: starting msgr at 2024-03-29T11:20:20.165 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.182+0000 7f92f876b640 1 librados: starting objecter 2024-03-29T11:20:20.165 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.182+0000 7f92f876b640 1 Processor -- start 2024-03-29T11:20:20.165 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.182+0000 7f92f876b640 1 -- start start 2024-03-29T11:20:20.166 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.182+0000 7f92f876b640 1 librados: setting wanted keys 2024-03-29T11:20:20.166 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.182+0000 7f92f876b640 1 librados: calling monclient init 2024-03-29T11:20:20.166 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.182+0000 7f92f876b640 10 monclient: init 2024-03-29T11:20:20.166 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.183+0000 7f92f876b640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:20.166 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.183+0000 7f92f876b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:20.166 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.183+0000 7f92f876b640 10 monclient(hunting): picked mon.noname-c con 0x7f92f00a9d70 addr v1:172.21.15.17:6790/0 2024-03-29T11:20:20.166 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.183+0000 7f92f876b640 10 monclient(hunting): picked mon.noname-a con 0x7f92f013f650 addr v1:172.21.15.17:6789/0 2024-03-29T11:20:20.166 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.183+0000 7f92f876b640 10 monclient(hunting): picked mon.noname-b con 0x7f92f01457f0 addr v1:172.21.15.94:6789/0 2024-03-29T11:20:20.166 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.184+0000 7f92f876b640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f00aced0 con 0x7f92f013f650 2024-03-29T11:20:20.166 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.184+0000 7f92f876b640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f01da5c0 con 0x7f92f01457f0 2024-03-29T11:20:20.167 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.184+0000 7f92f876b640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f01db1f0 con 0x7f92f00a9d70 2024-03-29T11:20:20.167 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.184+0000 7f92f876b640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:20.167 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.184+0000 7f92f876b640 10 monclient(hunting): authenticate will time out at 926.388912s 2024-03-29T11:20:20.167 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.184+0000 7f92f6507640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f92f00a9d70 0x7f92f01427e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:44870/0 (socket says 172.21.15.17:44870) 2024-03-29T11:20:20.167 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.184+0000 7f92f6507640 1 -- 172.21.15.17:0/1835240842 learned_addr learned my addr 172.21.15.17:0/1835240842 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:20:20.167 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.184+0000 7f92e6ffd640 10 client.?.objecter ms_handle_connect 0x7f92f00a9d70 2024-03-29T11:20:20.167 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.184+0000 7f92e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:20.167 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.184+0000 7f92e6ffd640 10 client.?.objecter ms_handle_connect 0x7f92f01457f0 2024-03-29T11:20:20.167 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.184+0000 7f92e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:20.167 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.185+0000 7f92e6ffd640 1 -- 172.21.15.17:0/1835240842 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3925216934 0 0) 0x7f92f01db1f0 con 0x7f92f00a9d70 2024-03-29T11:20:20.167 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.185+0000 7f92e6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:20.167 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.185+0000 7f92e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:20.167 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.185+0000 7f92e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:20.168 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.185+0000 7f92e6ffd640 10 monclient(hunting): my global_id is 4283 2024-03-29T11:20:20.168 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.185+0000 7f92e6ffd640 1 -- 172.21.15.17:0/1835240842 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92d4004eb0 con 0x7f92f00a9d70 2024-03-29T11:20:20.168 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.185+0000 7f92e6ffd640 1 -- 172.21.15.17:0/1835240842 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3747161068 0 0) 0x7f92d4004eb0 con 0x7f92f00a9d70 2024-03-29T11:20:20.168 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.185+0000 7f92e6ffd640 1 -- 172.21.15.17:0/1835240842 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92f01db1f0 con 0x7f92f00a9d70 2024-03-29T11:20:20.168 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.185+0000 7f92e6ffd640 1 -- 172.21.15.17:0/1835240842 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f92ec0017d0 con 0x7f92f00a9d70 2024-03-29T11:20:20.168 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.185+0000 7f92e6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:20.168 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.185+0000 7f92e6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:20.168 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.185+0000 7f92e6ffd640 10 monclient(hunting): dump: 2024-03-29T11:20:20.168 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:20:20.168 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:20:20.168 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:20.168 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:20.168 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:20.168 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:20.169 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:20:20.169 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:20:20.169 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:20:20.169 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:20.169 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.185+0000 7f92e6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:20.169 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.185+0000 7f92e6ffd640 1 -- 172.21.15.17:0/1835240842 <== mon.2 v1:172.21.15.17:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1912420603 0 0) 0x7f92f01db1f0 con 0x7f92f00a9d70 2024-03-29T11:20:20.169 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.185+0000 7f92e6ffd640 1 -- 172.21.15.17:0/1835240842 >> v1:172.21.15.94:6789/0 conn(0x7f92f01457f0 legacy=0x7f92f01e19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:20.169 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.185+0000 7f92e6ffd640 1 -- 172.21.15.17:0/1835240842 >> v1:172.21.15.17:6789/0 conn(0x7f92f013f650 legacy=0x7f92f01daae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:20:20.169 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.187+0000 7f92e6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:20.169 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.187+0000 7f92e6ffd640 1 monclient: found mon.c 2024-03-29T11:20:20.169 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.187+0000 7f92e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:20.169 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.187+0000 7f92e6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:20:20.169 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.187+0000 7f92e6ffd640 1 -- 172.21.15.17:0/1835240842 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f01db380 con 0x7f92f00a9d70 2024-03-29T11:20:20.170 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.187+0000 7f92e6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:20:20.170 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.187+0000 7f92e6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:20.170 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.187+0000 7f92e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:20.170 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.187+0000 7f92f876b640 5 monclient: authenticate success, global_id 4283 2024-03-29T11:20:20.170 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.187+0000 7f92f876b640 10 monclient: _renew_subs 2024-03-29T11:20:20.170 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.187+0000 7f92f876b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:20:20.170 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.187+0000 7f92f876b640 1 -- 172.21.15.17:0/1835240842 --> v1:172.21.15.17:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92f00ab620 con 0x7f92f00a9d70 2024-03-29T11:20:20.170 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.187+0000 7f92f876b640 10 client.4283.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:20.170 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.187+0000 7f92f876b640 10 monclient: _renew_subs 2024-03-29T11:20:20.170 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.187+0000 7f92f876b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:20:20.170 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.187+0000 7f92f876b640 1 -- 172.21.15.17:0/1835240842 --> v1:172.21.15.17:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f92f013ff80 con 0x7f92f00a9d70 2024-03-29T11:20:20.170 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.187+0000 7f92f876b640 1 librados: init done 2024-03-29T11:20:20.170 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.187+0000 7f92e6ffd640 1 -- 172.21.15.17:0/1835240842 <== mon.2 v1:172.21.15.17:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92ec001d60 con 0x7f92f00a9d70 2024-03-29T11:20:20.170 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.187+0000 7f92e6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:20.171 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.187+0000 7f92e6ffd640 1 -- 172.21.15.17:0/1835240842 <== mon.2 v1:172.21.15.17:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f92ec004860 con 0x7f92f00a9d70 2024-03-29T11:20:20.171 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.187+0000 7f92e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:20.171 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.187+0000 7f92e6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:20.171 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.187+0000 7f92e6ffd640 10 monclient: dump: 2024-03-29T11:20:20.171 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:20:20.171 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:20:20.171 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:20.171 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:20.171 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:20.171 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:20.171 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:20:20.171 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:20:20.171 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:20:20.172 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:20.172 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.187+0000 7f92f4d04640 4 set_mon_vals no callback set 2024-03-29T11:20:20.173 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.188+0000 7f92f876b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:20.173 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.188+0000 7f92f876b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:20.173 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.188+0000 7f92f876b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:20:20.173 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.188+0000 7f92f876b640 1 -- 172.21.15.17:0/1835240842 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92d8005080 con 0x7f92f00a9d70 2024-03-29T11:20:20.173 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.189+0000 7f92e6ffd640 1 -- 172.21.15.17:0/1835240842 <== mon.2 v1:172.21.15.17:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f92ec01a830 con 0x7f92f00a9d70 2024-03-29T11:20:20.173 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.189+0000 7f92e6ffd640 1 -- 172.21.15.17:0/1835240842 <== mon.2 v1:172.21.15.17:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f92ec084890 con 0x7f92f00a9d70 2024-03-29T11:20:20.173 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.189+0000 7f92e6ffd640 10 client.4283.objecter ms_dispatch 0x7f92f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:20.173 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.189+0000 7f92e6ffd640 3 client.4283.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:20.173 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.189+0000 7f92e6ffd640 3 client.4283.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:20.173 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.189+0000 7f92e6ffd640 20 client.4283.objecter dump_active .. 0 homeless 2024-03-29T11:20:20.173 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.189+0000 7f92f5d06640 10 monclient: get_auth_request con 0x7f92d406de00 auth_method 0 2024-03-29T11:20:20.173 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.190+0000 7f92e6ffd640 10 client.4283.objecter ms_handle_connect 0x7f92d406de00 2024-03-29T11:20:20.174 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.197+0000 7f92e6ffd640 1 -- 172.21.15.17:0/1835240842 <== mon.2 v1:172.21.15.17:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f92ec04ffa0 con 0x7f92f00a9d70 2024-03-29T11:20:20.174 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.197+0000 7f92e6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:20.174 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.197+0000 7f92e6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:20.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.523+0000 7f92f876b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:20.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.523+0000 7f92f876b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:20.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.523+0000 7f92f876b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:20:20.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.523+0000 7f92f876b640 1 -- 172.21.15.17:0/1835240842 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92d8005330 con 0x7f92f00a9d70 2024-03-29T11:20:20.501 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.525+0000 7f92e6ffd640 1 -- 172.21.15.17:0/1835240842 <== mon.2 v1:172.21.15.17:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f92ec0520a0 con 0x7f92f00a9d70 2024-03-29T11:20:20.501 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.525+0000 7f92e6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:20.501 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.525+0000 7f92e6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:20.501 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:20:20.502 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:20.505 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.528+0000 7f92f876b640 10 librados: watch_flush enter 2024-03-29T11:20:20.506 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.528+0000 7f92f876b640 10 librados: watch_flush exit 2024-03-29T11:20:20.506 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.528+0000 7f92f876b640 20 client.4283.objecter shutdown clearing up homeless session... 2024-03-29T11:20:20.506 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.528+0000 7f92f876b640 10 client.4283.objecter successfully canceled tick 2024-03-29T11:20:20.506 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.528+0000 7f92f876b640 1 -- 172.21.15.17:0/1835240842 >> v1:172.21.15.17:6816/3236185052 conn(0x7f92d406de00 legacy=0x7f92d40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:20.506 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.528+0000 7f92f876b640 10 monclient: shutdown 2024-03-29T11:20:20.506 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.528+0000 7f92f876b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:20.506 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.528+0000 7f92f876b640 1 -- 172.21.15.17:0/1835240842 >> v1:172.21.15.17:6790/0 conn(0x7f92f00a9d70 legacy=0x7f92f01427e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:20.506 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.529+0000 7f92f876b640 1 -- 172.21.15.17:0/1835240842 shutdown_connections 2024-03-29T11:20:20.506 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.529+0000 7f92f876b640 1 -- 172.21.15.17:0/1835240842 >> 172.21.15.17:0/1835240842 conn(0x7f92f0066a00 msgr2=0x7f92f00b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:20.506 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.529+0000 7f92f876b640 1 -- 172.21.15.17:0/1835240842 shutdown_connections 2024-03-29T11:20:20.506 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.529+0000 7f92f876b640 1 -- 172.21.15.17:0/1835240842 wait complete. 2024-03-29T11:20:20.506 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:20.529+0000 7f92f876b640 1 librados: shutdown 2024-03-29T11:20:20.522 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:27.329 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:27.477 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.500+0000 7f433db96640 10 monclient: get_monmap_and_config 2024-03-29T11:20:27.477 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.500+0000 7f433db96640 10 monclient: build_initial_monmap 2024-03-29T11:20:27.477 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.500+0000 7f433db96640 10 monclient: monmap: 2024-03-29T11:20:27.477 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:20:27.477 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:20:27.477 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:20:27.501394+0000 2024-03-29T11:20:27.477 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:20:27.501394+0000 2024-03-29T11:20:27.477 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:27.478 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:27.478 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:20:27.478 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:20:27.478 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:20:27.478 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:27.478 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.501+0000 7f433db96640 1 Processor -- start 2024-03-29T11:20:27.486 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.501+0000 7f433db96640 1 -- start start 2024-03-29T11:20:27.486 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.501+0000 7f433db96640 10 monclient: init 2024-03-29T11:20:27.486 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.502+0000 7f433db96640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:27.486 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.502+0000 7f433db96640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:27.487 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.502+0000 7f433db96640 10 monclient(hunting): picked mon.noname-b con 0x7f43380b0860 addr v1:172.21.15.17:6790/0 2024-03-29T11:20:27.487 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.502+0000 7f433db96640 10 monclient(hunting): picked mon.noname-a con 0x7f433813ef30 addr v1:172.21.15.17:6789/0 2024-03-29T11:20:27.487 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.502+0000 7f433db96640 10 monclient(hunting): picked mon.noname-c con 0x7f43380af8e0 addr v1:172.21.15.94:6789/0 2024-03-29T11:20:27.487 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.502+0000 7f433db96640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43380b0c40 con 0x7f433813ef30 2024-03-29T11:20:27.487 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.502+0000 7f433db96640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43380aed10 con 0x7f43380af8e0 2024-03-29T11:20:27.487 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.502+0000 7f433db96640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43380afd50 con 0x7f43380b0860 2024-03-29T11:20:27.487 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.502+0000 7f433db96640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:27.487 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.502+0000 7f433db96640 10 monclient(hunting): authenticate will time out at 933.707344s 2024-03-29T11:20:27.487 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.504+0000 7f4337fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f43380af8e0 0x7f4338149a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:55488/0 (socket says 172.21.15.17:55488) 2024-03-29T11:20:27.487 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.505+0000 7f4337fff640 1 -- 172.21.15.17:0/1505138426 learned_addr learned my addr 172.21.15.17:0/1505138426 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:20:27.487 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.506+0000 7f43367fc640 1 -- 172.21.15.17:0/1505138426 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1896474856 0 0) 0x7f43380b0c40 con 0x7f433813ef30 2024-03-29T11:20:27.487 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.506+0000 7f43367fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:27.487 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.506+0000 7f43367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:27.487 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.506+0000 7f43367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:27.488 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.506+0000 7f43367fc640 10 monclient(hunting): my global_id is 4272 2024-03-29T11:20:27.488 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.506+0000 7f43367fc640 1 -- 172.21.15.17:0/1505138426 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f432c003520 con 0x7f433813ef30 2024-03-29T11:20:27.488 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.506+0000 7f43367fc640 1 -- 172.21.15.17:0/1505138426 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2076985585 0 0) 0x7f43380aed10 con 0x7f43380af8e0 2024-03-29T11:20:27.488 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.506+0000 7f43367fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:27.488 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.506+0000 7f43367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:27.488 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.506+0000 7f43367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:27.488 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.506+0000 7f43367fc640 10 monclient(hunting): my global_id is 4294 2024-03-29T11:20:27.488 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.506+0000 7f43367fc640 1 -- 172.21.15.17:0/1505138426 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43380b0c40 con 0x7f43380af8e0 2024-03-29T11:20:27.488 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.506+0000 7f43367fc640 1 -- 172.21.15.17:0/1505138426 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 102811906 0 0) 0x7f43380b0c40 con 0x7f43380af8e0 2024-03-29T11:20:27.488 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.506+0000 7f43367fc640 1 -- 172.21.15.17:0/1505138426 >> v1:172.21.15.17:6790/0 conn(0x7f43380b0860 legacy=0x7f433813e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:27.488 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.506+0000 7f43367fc640 1 -- 172.21.15.17:0/1505138426 >> v1:172.21.15.17:6789/0 conn(0x7f433813ef30 legacy=0x7f43380af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:27.488 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.506+0000 7f43367fc640 10 monclient: _finish_hunting 0 2024-03-29T11:20:27.488 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.506+0000 7f43367fc640 1 monclient: found mon.noname-c 2024-03-29T11:20:27.488 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.506+0000 7f43367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:27.488 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.506+0000 7f43367fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6789/0 2024-03-29T11:20:27.489 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.506+0000 7f43367fc640 1 -- 172.21.15.17:0/1505138426 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43380ab1b0 con 0x7f43380af8e0 2024-03-29T11:20:27.489 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.507+0000 7f43367fc640 10 monclient: _finish_auth 0 2024-03-29T11:20:27.489 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.507+0000 7f43367fc640 10 monclient: _check_auth_tickets 2024-03-29T11:20:27.489 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.507+0000 7f43367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:27.489 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.507+0000 7f43367fc640 1 -- 172.21.15.17:0/1505138426 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f43300019f0 con 0x7f43380af8e0 2024-03-29T11:20:27.489 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.507+0000 7f43367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:27.489 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.507+0000 7f43367fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:27.489 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.507+0000 7f43367fc640 10 monclient: dump: 2024-03-29T11:20:27.489 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:20:27.489 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:20:27.489 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:27.489 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:27.489 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:27.489 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:27.489 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:20:27.490 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:20:27.490 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:20:27.490 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:27.490 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.507+0000 7f43367fc640 1 -- 172.21.15.17:0/1505138426 <== mon.1 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4330001b50 con 0x7f43380af8e0 2024-03-29T11:20:27.490 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.507+0000 7f43367fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:27.490 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.507+0000 7f43367fc640 1 -- 172.21.15.17:0/1505138426 <== mon.1 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f4330001f20 con 0x7f43380af8e0 2024-03-29T11:20:27.490 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.507+0000 7f43367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:27.490 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.507+0000 7f43367fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:27.490 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.507+0000 7f43367fc640 10 monclient: dump: 2024-03-29T11:20:27.490 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:20:27.490 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:20:27.490 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:27.490 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:27.490 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:27.491 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:27.491 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:20:27.491 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:20:27.491 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:20:27.491 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:27.491 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.507+0000 7f433db96640 5 monclient: authenticate success, global_id 4294 2024-03-29T11:20:27.491 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.507+0000 7f433db96640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:27.491 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.507+0000 7f433db96640 4 set_mon_vals no callback set 2024-03-29T11:20:27.491 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.508+0000 7f433db96640 10 monclient: shutdown 2024-03-29T11:20:27.491 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.508+0000 7f433db96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:27.491 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.508+0000 7f433db96640 1 -- 172.21.15.17:0/1505138426 >> v1:172.21.15.94:6789/0 conn(0x7f43380af8e0 legacy=0x7f4338149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:27.491 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.509+0000 7f433db96640 1 -- 172.21.15.17:0/1505138426 shutdown_connections 2024-03-29T11:20:27.491 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.509+0000 7f433db96640 1 -- 172.21.15.17:0/1505138426 >> 172.21.15.17:0/1505138426 conn(0x7f433806b660 msgr2=0x7f433806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:27.491 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.509+0000 7f433db96640 1 -- 172.21.15.17:0/1505138426 shutdown_connections 2024-03-29T11:20:27.491 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.510+0000 7f433db96640 1 -- 172.21.15.17:0/1505138426 wait complete. 2024-03-29T11:20:27.492 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.511+0000 7f433db96640 10 monclient: build_initial_monmap 2024-03-29T11:20:27.492 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.511+0000 7f433db96640 10 monclient: monmap: 2024-03-29T11:20:27.492 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:20:27.492 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:27.492 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:20:27.492 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:20:27.492 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:27.492 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:27.493 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:20:27.493 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:20:27.493 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:20:27.493 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:27.493 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.511+0000 7f433db96640 1 librados: starting msgr at 2024-03-29T11:20:27.493 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.511+0000 7f433db96640 1 librados: starting objecter 2024-03-29T11:20:27.493 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.512+0000 7f433db96640 1 Processor -- start 2024-03-29T11:20:27.493 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.512+0000 7f433db96640 1 -- start start 2024-03-29T11:20:27.493 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.512+0000 7f433db96640 1 librados: setting wanted keys 2024-03-29T11:20:27.493 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.512+0000 7f433db96640 1 librados: calling monclient init 2024-03-29T11:20:27.493 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.512+0000 7f433db96640 10 monclient: init 2024-03-29T11:20:27.493 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.512+0000 7f433db96640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:27.494 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.513+0000 7f433db96640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:20:27.494 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.513+0000 7f433db96640 10 monclient(hunting): picked mon.noname-c con 0x7f43380af8e0 addr v1:172.21.15.17:6790/0 2024-03-29T11:20:27.494 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.513+0000 7f433db96640 10 monclient(hunting): picked mon.noname-b con 0x7f43380b0860 addr v1:172.21.15.94:6789/0 2024-03-29T11:20:27.494 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.513+0000 7f433db96640 10 monclient(hunting): picked mon.noname-a con 0x7f433813ef30 addr v1:172.21.15.17:6789/0 2024-03-29T11:20:27.494 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.513+0000 7f433db96640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43380b2570 con 0x7f433813ef30 2024-03-29T11:20:27.494 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.513+0000 7f433db96640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4338135ab0 con 0x7f43380b0860 2024-03-29T11:20:27.494 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.513+0000 7f433db96640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43381db9a0 con 0x7f43380af8e0 2024-03-29T11:20:27.494 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.513+0000 7f433db96640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:27.494 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.513+0000 7f433db96640 10 monclient(hunting): authenticate will time out at 933.717971s 2024-03-29T11:20:27.494 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.513+0000 7f4336ffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f43380b0860 0x7f43380b3160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:55984/0 (socket says 172.21.15.17:55984) 2024-03-29T11:20:27.494 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.513+0000 7f4336ffd640 1 -- 172.21.15.17:0/3752471421 learned_addr learned my addr 172.21.15.17:0/3752471421 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:20:27.499 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.517+0000 7f4327fff640 10 client.?.objecter ms_handle_connect 0x7f43380b0860 2024-03-29T11:20:27.499 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.517+0000 7f4327fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:27.499 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.517+0000 7f4327fff640 10 client.?.objecter ms_handle_connect 0x7f433813ef30 2024-03-29T11:20:27.499 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.517+0000 7f4327fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:27.499 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.517+0000 7f4327fff640 1 -- 172.21.15.17:0/3752471421 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3226468000 0 0) 0x7f4338135ab0 con 0x7f43380b0860 2024-03-29T11:20:27.499 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.517+0000 7f4327fff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:27.499 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.517+0000 7f4327fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:27.499 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.517+0000 7f4327fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:27.499 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.517+0000 7f4327fff640 10 monclient(hunting): my global_id is 4297 2024-03-29T11:20:27.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.517+0000 7f4327fff640 1 -- 172.21.15.17:0/3752471421 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4318004eb0 con 0x7f43380b0860 2024-03-29T11:20:27.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.517+0000 7f4327fff640 1 -- 172.21.15.17:0/3752471421 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2739973645 0 0) 0x7f43380b2570 con 0x7f433813ef30 2024-03-29T11:20:27.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.517+0000 7f4327fff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:27.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.517+0000 7f4327fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:27.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.518+0000 7f4327fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:27.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.518+0000 7f4327fff640 10 monclient(hunting): my global_id is 4275 2024-03-29T11:20:27.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.518+0000 7f4327fff640 1 -- 172.21.15.17:0/3752471421 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4338135ab0 con 0x7f433813ef30 2024-03-29T11:20:27.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.518+0000 7f4327fff640 1 -- 172.21.15.17:0/3752471421 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 166484791 0 0) 0x7f4318004eb0 con 0x7f43380b0860 2024-03-29T11:20:27.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.518+0000 7f4327fff640 1 -- 172.21.15.17:0/3752471421 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43380b2570 con 0x7f43380b0860 2024-03-29T11:20:27.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.518+0000 7f4327fff640 1 -- 172.21.15.17:0/3752471421 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f4320001970 con 0x7f43380b0860 2024-03-29T11:20:27.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.518+0000 7f4327fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:27.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.518+0000 7f4327fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:27.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.518+0000 7f4327fff640 10 monclient(hunting): dump: 2024-03-29T11:20:27.500 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:20:27.500 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:20:27.501 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:27.501 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:27.501 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:27.501 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:27.501 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:20:27.501 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:20:27.501 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:20:27.501 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:27.501 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.518+0000 7f4327fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:27.501 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.518+0000 7f4327fff640 1 -- 172.21.15.17:0/3752471421 <== mon.1 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2934403479 0 0) 0x7f43380b2570 con 0x7f43380b0860 2024-03-29T11:20:27.501 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.518+0000 7f4327fff640 1 -- 172.21.15.17:0/3752471421 >> v1:172.21.15.17:6790/0 conn(0x7f43380af8e0 legacy=0x7f43380b2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:27.501 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.519+0000 7f4327fff640 1 -- 172.21.15.17:0/3752471421 >> v1:172.21.15.17:6789/0 conn(0x7f433813ef30 legacy=0x7f43381fff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:27.501 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.519+0000 7f4327fff640 10 monclient: _finish_hunting 0 2024-03-29T11:20:27.501 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.519+0000 7f4327fff640 1 monclient: found mon.b 2024-03-29T11:20:27.501 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.519+0000 7f4327fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:27.502 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.519+0000 7f4327fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:20:27.502 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.519+0000 7f4327fff640 1 -- 172.21.15.17:0/3752471421 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43381dbb30 con 0x7f43380b0860 2024-03-29T11:20:27.502 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.519+0000 7f4327fff640 10 monclient: _finish_auth 0 2024-03-29T11:20:27.502 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.519+0000 7f4327fff640 10 monclient: _check_auth_tickets 2024-03-29T11:20:27.502 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.519+0000 7f4327fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:27.502 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.519+0000 7f433db96640 5 monclient: authenticate success, global_id 4297 2024-03-29T11:20:27.502 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.519+0000 7f433db96640 10 monclient: _renew_subs 2024-03-29T11:20:27.502 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.519+0000 7f433db96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:20:27.502 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.519+0000 7f433db96640 1 -- 172.21.15.17:0/3752471421 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43381346c0 con 0x7f43380b0860 2024-03-29T11:20:27.502 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.519+0000 7f433db96640 10 client.4297.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:27.502 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.519+0000 7f433db96640 10 monclient: _renew_subs 2024-03-29T11:20:27.502 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.519+0000 7f433db96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:20:27.502 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.519+0000 7f433db96640 1 -- 172.21.15.17:0/3752471421 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f43381c8500 con 0x7f43380b0860 2024-03-29T11:20:27.502 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.519+0000 7f433db96640 1 librados: init done 2024-03-29T11:20:27.503 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.520+0000 7f4327fff640 1 -- 172.21.15.17:0/3752471421 <== mon.1 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43200035f0 con 0x7f43380b0860 2024-03-29T11:20:27.503 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.520+0000 7f4327fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:27.503 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.520+0000 7f4327fff640 1 -- 172.21.15.17:0/3752471421 <== mon.1 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f4320003940 con 0x7f43380b0860 2024-03-29T11:20:27.503 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.520+0000 7f4327fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:27.503 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.520+0000 7f4327fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:27.503 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.520+0000 7f4335ffb640 4 set_mon_vals no callback set 2024-03-29T11:20:27.503 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.520+0000 7f4327fff640 10 monclient: dump: 2024-03-29T11:20:27.503 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:20:27.503 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:20:27.503 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:27.503 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:27.503 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:27.503 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:27.503 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:20:27.503 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:20:27.504 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:20:27.504 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:27.504 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.522+0000 7f4327fff640 1 -- 172.21.15.17:0/3752471421 <== mon.1 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f4320003aa0 con 0x7f43380b0860 2024-03-29T11:20:27.505 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.523+0000 7f4327fff640 1 -- 172.21.15.17:0/3752471421 <== mon.1 v1:172.21.15.94:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f4320084a20 con 0x7f43380b0860 2024-03-29T11:20:27.505 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.523+0000 7f4327fff640 10 client.4297.objecter ms_dispatch 0x7f4338000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:27.505 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.523+0000 7f4327fff640 3 client.4297.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:27.505 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.523+0000 7f4327fff640 3 client.4297.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:27.505 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.523+0000 7f4327fff640 20 client.4297.objecter dump_active .. 0 homeless 2024-03-29T11:20:27.505 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.523+0000 7f43377fe640 10 monclient: get_auth_request con 0x7f431806dba0 auth_method 0 2024-03-29T11:20:27.505 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.523+0000 7f433db96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:27.505 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.523+0000 7f433db96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:27.505 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.523+0000 7f433db96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:20:27.505 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.523+0000 7f433db96640 1 -- 172.21.15.17:0/3752471421 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f431c005080 con 0x7f43380b0860 2024-03-29T11:20:27.505 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.524+0000 7f4327fff640 10 client.4297.objecter ms_handle_connect 0x7f431806dba0 2024-03-29T11:20:27.509 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.533+0000 7f4327fff640 1 -- 172.21.15.17:0/3752471421 <== mon.1 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4320050300 con 0x7f43380b0860 2024-03-29T11:20:27.510 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.533+0000 7f4327fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:27.510 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.533+0000 7f4327fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:27.831 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.855+0000 7f433db96640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:27.832 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.855+0000 7f433db96640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:27.832 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.855+0000 7f433db96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:20:27.832 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.855+0000 7f433db96640 1 -- 172.21.15.17:0/3752471421 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f431c005330 con 0x7f43380b0860 2024-03-29T11:20:27.833 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.856+0000 7f4327fff640 1 -- 172.21.15.17:0/3752471421 <== mon.1 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f4320052230 con 0x7f43380b0860 2024-03-29T11:20:27.833 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.856+0000 7f4327fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:27.833 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.856+0000 7f4327fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:27.834 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:20:27.834 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:27.839 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.860+0000 7f433db96640 10 librados: watch_flush enter 2024-03-29T11:20:27.839 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.861+0000 7f433db96640 10 librados: watch_flush exit 2024-03-29T11:20:27.839 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.861+0000 7f433db96640 20 client.4297.objecter shutdown clearing up homeless session... 2024-03-29T11:20:27.839 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.861+0000 7f433db96640 10 client.4297.objecter successfully canceled tick 2024-03-29T11:20:27.839 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.862+0000 7f433db96640 1 -- 172.21.15.17:0/3752471421 >> v1:172.21.15.17:6816/3236185052 conn(0x7f431806dba0 legacy=0x7f4318070060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:27.839 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.862+0000 7f433db96640 10 monclient: shutdown 2024-03-29T11:20:27.839 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.862+0000 7f433db96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:27.839 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.862+0000 7f433db96640 1 -- 172.21.15.17:0/3752471421 >> v1:172.21.15.94:6789/0 conn(0x7f43380b0860 legacy=0x7f43380b3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:27.839 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.862+0000 7f433db96640 1 -- 172.21.15.17:0/3752471421 shutdown_connections 2024-03-29T11:20:27.839 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.862+0000 7f433db96640 1 -- 172.21.15.17:0/3752471421 >> 172.21.15.17:0/3752471421 conn(0x7f433806b660 msgr2=0x7f43380adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:27.839 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.862+0000 7f433db96640 1 -- 172.21.15.17:0/3752471421 shutdown_connections 2024-03-29T11:20:27.840 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.862+0000 7f433db96640 1 -- 172.21.15.17:0/3752471421 wait complete. 2024-03-29T11:20:27.840 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:27.862+0000 7f433db96640 1 librados: shutdown 2024-03-29T11:20:27.856 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:34.663 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:34.817 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.838+0000 7f4611484640 10 monclient: get_monmap_and_config 2024-03-29T11:20:34.817 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.838+0000 7f4611484640 10 monclient: build_initial_monmap 2024-03-29T11:20:34.818 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.838+0000 7f4611484640 10 monclient: monmap: 2024-03-29T11:20:34.818 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:20:34.818 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:20:34.818 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:20:34.839402+0000 2024-03-29T11:20:34.818 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:20:34.839402+0000 2024-03-29T11:20:34.818 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:34.818 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:34.818 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:20:34.818 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:20:34.818 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:20:34.818 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:34.818 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.839+0000 7f4611484640 1 Processor -- start 2024-03-29T11:20:34.818 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.840+0000 7f4611484640 1 -- start start 2024-03-29T11:20:34.818 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.840+0000 7f4611484640 10 monclient: init 2024-03-29T11:20:34.819 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.840+0000 7f4611484640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:34.819 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.840+0000 7f4611484640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:20:34.819 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.840+0000 7f4611484640 10 monclient(hunting): picked mon.noname-b con 0x7f460c06c250 addr v1:172.21.15.17:6790/0 2024-03-29T11:20:34.819 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.840+0000 7f4611484640 10 monclient(hunting): picked mon.noname-c con 0x7f460c072400 addr v1:172.21.15.94:6789/0 2024-03-29T11:20:34.819 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.840+0000 7f4611484640 10 monclient(hunting): picked mon.noname-a con 0x7f460c06c630 addr v1:172.21.15.17:6789/0 2024-03-29T11:20:34.819 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.840+0000 7f4611484640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f460c06d760 con 0x7f460c06c630 2024-03-29T11:20:34.819 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.840+0000 7f4611484640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f460c070500 con 0x7f460c072400 2024-03-29T11:20:34.819 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.840+0000 7f4611484640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f460c0aff50 con 0x7f460c06c250 2024-03-29T11:20:34.819 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.840+0000 7f4611484640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:34.819 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.840+0000 7f4611484640 10 monclient(hunting): authenticate will time out at 941.045800s 2024-03-29T11:20:34.820 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.843+0000 7f460b7fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f460c06c630 0x7f460c151ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:52589/0 (socket says 172.21.15.17:52589) 2024-03-29T11:20:34.820 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.844+0000 7f460b7fe640 1 -- 172.21.15.17:0/965407401 learned_addr learned my addr 172.21.15.17:0/965407401 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:20:34.822 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.844+0000 7f4609ffb640 1 -- 172.21.15.17:0/965407401 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2417542490 0 0) 0x7f460c0aff50 con 0x7f460c06c250 2024-03-29T11:20:34.822 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.844+0000 7f4609ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:34.822 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.844+0000 7f4609ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:34.822 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.844+0000 7f4609ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:34.822 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.844+0000 7f4609ffb640 10 monclient(hunting): my global_id is 4289 2024-03-29T11:20:34.822 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.844+0000 7f4609ffb640 1 -- 172.21.15.17:0/965407401 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45f4003520 con 0x7f460c06c250 2024-03-29T11:20:34.822 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.845+0000 7f4609ffb640 1 -- 172.21.15.17:0/965407401 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2197031055 0 0) 0x7f460c06d760 con 0x7f460c06c630 2024-03-29T11:20:34.822 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.845+0000 7f4609ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:34.822 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.845+0000 7f4609ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:34.822 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.845+0000 7f4609ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:34.822 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.845+0000 7f4609ffb640 10 monclient(hunting): my global_id is 4278 2024-03-29T11:20:34.822 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.845+0000 7f4609ffb640 1 -- 172.21.15.17:0/965407401 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f460c0aff50 con 0x7f460c06c630 2024-03-29T11:20:34.823 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.846+0000 7f4609ffb640 1 -- 172.21.15.17:0/965407401 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 21969887 0 0) 0x7f460c070500 con 0x7f460c072400 2024-03-29T11:20:34.823 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.846+0000 7f4609ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:34.824 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.846+0000 7f4609ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:34.824 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.846+0000 7f4609ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:34.824 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.846+0000 7f4609ffb640 10 monclient(hunting): my global_id is 4300 2024-03-29T11:20:34.824 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.846+0000 7f4609ffb640 1 -- 172.21.15.17:0/965407401 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f460c06d760 con 0x7f460c072400 2024-03-29T11:20:34.824 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.847+0000 7f4609ffb640 1 -- 172.21.15.17:0/965407401 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3942901921 0 0) 0x7f460c06d760 con 0x7f460c072400 2024-03-29T11:20:34.824 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.847+0000 7f4609ffb640 1 -- 172.21.15.17:0/965407401 >> v1:172.21.15.17:6790/0 conn(0x7f460c06c250 legacy=0x7f460c071cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:34.824 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.847+0000 7f4609ffb640 1 -- 172.21.15.17:0/965407401 >> v1:172.21.15.17:6789/0 conn(0x7f460c06c630 legacy=0x7f460c151ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:34.825 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.848+0000 7f4609ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:20:34.825 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.848+0000 7f4609ffb640 1 monclient: found mon.noname-c 2024-03-29T11:20:34.825 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.848+0000 7f4609ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:34.825 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.848+0000 7f4609ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6789/0 2024-03-29T11:20:34.825 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.848+0000 7f4609ffb640 1 -- 172.21.15.17:0/965407401 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f460c0b00e0 con 0x7f460c072400 2024-03-29T11:20:34.825 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.848+0000 7f4609ffb640 10 monclient: _finish_auth 0 2024-03-29T11:20:34.825 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.848+0000 7f4609ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:20:34.825 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.848+0000 7f4609ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:34.825 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.848+0000 7f4609ffb640 1 -- 172.21.15.17:0/965407401 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f45fc001570 con 0x7f460c072400 2024-03-29T11:20:34.825 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.848+0000 7f4609ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:34.825 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.848+0000 7f4609ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:34.825 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.848+0000 7f4609ffb640 10 monclient: dump: 2024-03-29T11:20:34.826 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:20:34.826 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:20:34.826 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:34.826 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:34.826 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:34.826 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:34.826 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:20:34.826 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:20:34.826 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:20:34.826 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:34.826 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.848+0000 7f4609ffb640 1 -- 172.21.15.17:0/965407401 <== mon.1 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45fc001800 con 0x7f460c072400 2024-03-29T11:20:34.826 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.848+0000 7f4609ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:34.826 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.848+0000 7f4609ffb640 1 -- 172.21.15.17:0/965407401 <== mon.1 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f45fc001b90 con 0x7f460c072400 2024-03-29T11:20:34.826 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.848+0000 7f4609ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:34.827 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.848+0000 7f4609ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:34.827 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.848+0000 7f4609ffb640 10 monclient: dump: 2024-03-29T11:20:34.827 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:20:34.827 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:20:34.827 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:34.827 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:34.827 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:34.827 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:34.827 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:20:34.827 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:20:34.827 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:20:34.827 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:34.827 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.848+0000 7f4611484640 5 monclient: authenticate success, global_id 4300 2024-03-29T11:20:34.827 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.848+0000 7f4611484640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:34.827 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.848+0000 7f4611484640 4 set_mon_vals no callback set 2024-03-29T11:20:34.829 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.849+0000 7f4611484640 10 monclient: shutdown 2024-03-29T11:20:34.829 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.849+0000 7f4611484640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:34.829 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.849+0000 7f4611484640 1 -- 172.21.15.17:0/965407401 >> v1:172.21.15.94:6789/0 conn(0x7f460c072400 legacy=0x7f460c1465b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:34.829 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.849+0000 7f4611484640 1 -- 172.21.15.17:0/965407401 shutdown_connections 2024-03-29T11:20:34.829 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.849+0000 7f4611484640 1 -- 172.21.15.17:0/965407401 >> 172.21.15.17:0/965407401 conn(0x7f460c067610 msgr2=0x7f460c0b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:34.829 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.850+0000 7f4611484640 1 -- 172.21.15.17:0/965407401 shutdown_connections 2024-03-29T11:20:34.829 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.850+0000 7f4611484640 1 -- 172.21.15.17:0/965407401 wait complete. 2024-03-29T11:20:34.829 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.851+0000 7f4611484640 10 monclient: build_initial_monmap 2024-03-29T11:20:34.829 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.851+0000 7f4611484640 10 monclient: monmap: 2024-03-29T11:20:34.830 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:20:34.830 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:34.830 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:20:34.830 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:20:34.830 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:34.830 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:34.830 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:20:34.830 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:20:34.830 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:20:34.830 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:34.831 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.852+0000 7f4611484640 1 librados: starting msgr at 2024-03-29T11:20:34.831 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.852+0000 7f4611484640 1 librados: starting objecter 2024-03-29T11:20:34.831 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.852+0000 7f4611484640 1 Processor -- start 2024-03-29T11:20:34.831 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.852+0000 7f4611484640 1 -- start start 2024-03-29T11:20:34.831 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.852+0000 7f4611484640 1 librados: setting wanted keys 2024-03-29T11:20:34.831 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.852+0000 7f4611484640 1 librados: calling monclient init 2024-03-29T11:20:34.831 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.852+0000 7f4611484640 10 monclient: init 2024-03-29T11:20:34.832 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.853+0000 7f4611484640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:34.832 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.853+0000 7f4611484640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:20:34.832 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.853+0000 7f4611484640 10 monclient(hunting): picked mon.noname-b con 0x7f460c06c250 addr v1:172.21.15.94:6789/0 2024-03-29T11:20:34.832 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.854+0000 7f4611484640 10 monclient(hunting): picked mon.noname-c con 0x7f460c06c630 addr v1:172.21.15.17:6790/0 2024-03-29T11:20:34.832 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.854+0000 7f4611484640 10 monclient(hunting): picked mon.noname-a con 0x7f460c072400 addr v1:172.21.15.17:6789/0 2024-03-29T11:20:34.832 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.854+0000 7f4611484640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f460c0abb70 con 0x7f460c072400 2024-03-29T11:20:34.833 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.854+0000 7f4611484640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f460c0ac0c0 con 0x7f460c06c250 2024-03-29T11:20:34.833 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.855+0000 7f4611484640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f460c0abde0 con 0x7f460c06c630 2024-03-29T11:20:34.833 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.855+0000 7f4611484640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:34.833 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.855+0000 7f4611484640 10 monclient(hunting): authenticate will time out at 941.060409s 2024-03-29T11:20:34.843 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.860+0000 7f460affd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f460c06c250 0x7f460c13f2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:49469/0 (socket says 172.21.15.17:49469) 2024-03-29T11:20:34.844 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.860+0000 7f460b7fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f460c072400 0x7f460c1dc120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:52747/0 (socket says 172.21.15.17:52747) 2024-03-29T11:20:34.844 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.860+0000 7f460b7fe640 1 -- 172.21.15.17:0/3706602791 learned_addr learned my addr 172.21.15.17:0/3706602791 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:20:34.844 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.860+0000 7f45f37fe640 10 client.?.objecter ms_handle_connect 0x7f460c072400 2024-03-29T11:20:34.844 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.860+0000 7f45f37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:34.844 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.860+0000 7f45f37fe640 10 client.?.objecter ms_handle_connect 0x7f460c06c250 2024-03-29T11:20:34.844 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.860+0000 7f45f37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:34.844 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.860+0000 7f45f37fe640 1 -- 172.21.15.17:0/3706602791 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3772088887 0 0) 0x7f460c0ac0c0 con 0x7f460c06c250 2024-03-29T11:20:34.844 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.861+0000 7f45f37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:34.844 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.861+0000 7f45f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:34.844 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.861+0000 7f45f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:34.844 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.861+0000 7f45f37fe640 10 monclient(hunting): my global_id is 4303 2024-03-29T11:20:34.844 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.861+0000 7f45f37fe640 1 -- 172.21.15.17:0/3706602791 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45f8004eb0 con 0x7f460c06c250 2024-03-29T11:20:34.844 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.861+0000 7f45f37fe640 1 -- 172.21.15.17:0/3706602791 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 355958478 0 0) 0x7f460c0abb70 con 0x7f460c072400 2024-03-29T11:20:34.845 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.861+0000 7f45f37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:34.845 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.861+0000 7f45f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:34.845 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.861+0000 7f45f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:34.845 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.861+0000 7f45f37fe640 10 monclient(hunting): my global_id is 4281 2024-03-29T11:20:34.845 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.861+0000 7f45f37fe640 1 -- 172.21.15.17:0/3706602791 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f460c0ac0c0 con 0x7f460c072400 2024-03-29T11:20:34.845 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.861+0000 7f45f37fe640 1 -- 172.21.15.17:0/3706602791 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1257158603 0 0) 0x7f45f8004eb0 con 0x7f460c06c250 2024-03-29T11:20:34.845 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.861+0000 7f45f37fe640 1 -- 172.21.15.17:0/3706602791 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f460c0abb70 con 0x7f460c06c250 2024-03-29T11:20:34.845 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.862+0000 7f45f37fe640 1 -- 172.21.15.17:0/3706602791 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f4600001950 con 0x7f460c06c250 2024-03-29T11:20:34.845 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.862+0000 7f45f37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:34.846 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.862+0000 7f45f37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:34.846 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.862+0000 7f45f37fe640 10 monclient(hunting): dump: 2024-03-29T11:20:34.846 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:20:34.846 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:20:34.846 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:34.846 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:34.846 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:34.846 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:34.846 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:20:34.846 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:20:34.846 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:20:34.846 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:34.846 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.862+0000 7f45f37fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:34.846 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.862+0000 7f45f37fe640 10 client.?.objecter ms_handle_connect 0x7f460c06c630 2024-03-29T11:20:34.847 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.862+0000 7f45f37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:34.847 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.862+0000 7f45f37fe640 1 -- 172.21.15.17:0/3706602791 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3482133375 0 0) 0x7f460c0ac0c0 con 0x7f460c072400 2024-03-29T11:20:34.847 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.862+0000 7f45f37fe640 1 -- 172.21.15.17:0/3706602791 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45f8004eb0 con 0x7f460c072400 2024-03-29T11:20:34.847 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.862+0000 7f45f37fe640 1 -- 172.21.15.17:0/3706602791 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f4604001610 con 0x7f460c072400 2024-03-29T11:20:34.847 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.862+0000 7f45f37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:34.847 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.862+0000 7f45f37fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:34.847 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.862+0000 7f45f37fe640 10 monclient(hunting): dump: 2024-03-29T11:20:34.847 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:20:34.847 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:20:34.847 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:34.847 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:34.847 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:34.847 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:34.847 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:20:34.847 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:20:34.848 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:20:34.848 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:34.848 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.862+0000 7f45f37fe640 1 -- 172.21.15.17:0/3706602791 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3795678522 0 0) 0x7f460c0abde0 con 0x7f460c06c630 2024-03-29T11:20:34.848 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.862+0000 7f45f37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:34.848 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.862+0000 7f45f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:34.848 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.862+0000 7f45f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:34.848 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.862+0000 7f45f37fe640 10 monclient(hunting): my global_id is 4292 2024-03-29T11:20:34.849 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.862+0000 7f45f37fe640 1 -- 172.21.15.17:0/3706602791 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f460c0ac0c0 con 0x7f460c06c630 2024-03-29T11:20:34.849 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.862+0000 7f45f37fe640 1 -- 172.21.15.17:0/3706602791 <== mon.0 v1:172.21.15.17:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2534966745 0 0) 0x7f45f8004eb0 con 0x7f460c072400 2024-03-29T11:20:34.849 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.862+0000 7f45f37fe640 1 -- 172.21.15.17:0/3706602791 >> v1:172.21.15.17:6790/0 conn(0x7f460c06c630 legacy=0x7f460c1dba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:34.849 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.862+0000 7f45f37fe640 1 -- 172.21.15.17:0/3706602791 >> v1:172.21.15.94:6789/0 conn(0x7f460c06c250 legacy=0x7f460c13f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:34.849 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.862+0000 7f45f37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:20:34.849 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.862+0000 7f45f37fe640 1 monclient: found mon.a 2024-03-29T11:20:34.849 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.862+0000 7f45f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:34.849 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.862+0000 7f45f37fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:20:34.849 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.862+0000 7f45f37fe640 1 -- 172.21.15.17:0/3706602791 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f460c0aaf60 con 0x7f460c072400 2024-03-29T11:20:34.849 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.862+0000 7f45f37fe640 10 monclient: _finish_auth 0 2024-03-29T11:20:34.849 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.862+0000 7f45f37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:20:34.849 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.862+0000 7f45f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:34.850 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.862+0000 7f4611484640 5 monclient: authenticate success, global_id 4281 2024-03-29T11:20:34.850 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.862+0000 7f4611484640 10 monclient: _renew_subs 2024-03-29T11:20:34.850 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.862+0000 7f4611484640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:20:34.850 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.862+0000 7f4611484640 1 -- 172.21.15.17:0/3706602791 --> v1:172.21.15.17:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f460c0aa960 con 0x7f460c072400 2024-03-29T11:20:34.850 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.862+0000 7f4611484640 10 client.4281.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:34.850 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.862+0000 7f4611484640 10 monclient: _renew_subs 2024-03-29T11:20:34.850 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.862+0000 7f4611484640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:20:34.850 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.862+0000 7f4611484640 1 -- 172.21.15.17:0/3706602791 --> v1:172.21.15.17:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f460c13fa00 con 0x7f460c072400 2024-03-29T11:20:34.850 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.862+0000 7f4611484640 1 librados: init done 2024-03-29T11:20:34.851 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.864+0000 7f45f37fe640 1 -- 172.21.15.17:0/3706602791 <== mon.0 v1:172.21.15.17:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4604001a20 con 0x7f460c072400 2024-03-29T11:20:34.851 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.864+0000 7f45f37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:34.851 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.864+0000 7f45f37fe640 1 -- 172.21.15.17:0/3706602791 <== mon.0 v1:172.21.15.17:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f4604004d00 con 0x7f460c072400 2024-03-29T11:20:34.851 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.864+0000 7f46097fa640 4 set_mon_vals no callback set 2024-03-29T11:20:34.851 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.864+0000 7f45f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:34.851 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.864+0000 7f45f37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:34.851 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.865+0000 7f45f37fe640 10 monclient: dump: 2024-03-29T11:20:34.851 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:20:34.851 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:20:34.851 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:34.851 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:34.851 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:34.851 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:34.851 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:20:34.852 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:20:34.852 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:20:34.852 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:34.852 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.864+0000 7f4611484640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:34.852 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.865+0000 7f4611484640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:34.852 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.865+0000 7f4611484640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:20:34.852 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.865+0000 7f4611484640 1 -- 172.21.15.17:0/3706602791 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f460c075f10 con 0x7f460c072400 2024-03-29T11:20:34.853 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.869+0000 7f45f37fe640 1 -- 172.21.15.17:0/3706602791 <== mon.0 v1:172.21.15.17:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f460401acd0 con 0x7f460c072400 2024-03-29T11:20:34.853 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.870+0000 7f45f37fe640 1 -- 172.21.15.17:0/3706602791 <== mon.0 v1:172.21.15.17:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f4604084cd0 con 0x7f460c072400 2024-03-29T11:20:34.853 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.873+0000 7f460affd640 10 monclient: get_auth_request con 0x7f45f806e060 auth_method 0 2024-03-29T11:20:34.853 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.874+0000 7f45f37fe640 10 client.4281.objecter ms_dispatch 0x7f460c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:34.853 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.874+0000 7f45f37fe640 3 client.4281.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:34.853 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.874+0000 7f45f37fe640 3 client.4281.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:34.853 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.874+0000 7f45f37fe640 20 client.4281.objecter dump_active .. 0 homeless 2024-03-29T11:20:34.853 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.874+0000 7f45f37fe640 10 client.4281.objecter ms_handle_connect 0x7f45f806e060 2024-03-29T11:20:34.854 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.878+0000 7f45f37fe640 1 -- 172.21.15.17:0/3706602791 <== mon.0 v1:172.21.15.17:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f46040503e0 con 0x7f460c072400 2024-03-29T11:20:34.855 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.878+0000 7f45f37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:34.855 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:34.879+0000 7f45f37fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:35.198 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:35.222+0000 7f4611484640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:35.198 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:35.222+0000 7f4611484640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:35.198 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:35.222+0000 7f4611484640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:20:35.199 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:35.222+0000 7f4611484640 1 -- 172.21.15.17:0/3706602791 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f460c149040 con 0x7f460c072400 2024-03-29T11:20:35.201 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:35.224+0000 7f45f37fe640 1 -- 172.21.15.17:0/3706602791 <== mon.0 v1:172.21.15.17:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f46040524e0 con 0x7f460c072400 2024-03-29T11:20:35.201 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:35.224+0000 7f45f37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:35.201 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:35.224+0000 7f45f37fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:35.201 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:20:35.201 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:35.205 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:35.228+0000 7f4611484640 10 librados: watch_flush enter 2024-03-29T11:20:35.206 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:35.230+0000 7f4611484640 10 librados: watch_flush exit 2024-03-29T11:20:35.206 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:35.230+0000 7f4611484640 20 client.4281.objecter shutdown clearing up homeless session... 2024-03-29T11:20:35.206 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:35.230+0000 7f4611484640 10 client.4281.objecter successfully canceled tick 2024-03-29T11:20:35.207 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:35.230+0000 7f4611484640 1 -- 172.21.15.17:0/3706602791 >> v1:172.21.15.17:6816/3236185052 conn(0x7f45f806e060 legacy=0x7f45f8070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.207 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:35.230+0000 7f4611484640 10 monclient: shutdown 2024-03-29T11:20:35.207 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:35.230+0000 7f4611484640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:35.207 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:35.230+0000 7f4611484640 1 -- 172.21.15.17:0/3706602791 >> v1:172.21.15.17:6789/0 conn(0x7f460c072400 legacy=0x7f460c1dc120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.210 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:35.232+0000 7f4611484640 1 -- 172.21.15.17:0/3706602791 shutdown_connections 2024-03-29T11:20:35.210 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:35.232+0000 7f4611484640 1 -- 172.21.15.17:0/3706602791 >> 172.21.15.17:0/3706602791 conn(0x7f460c067610 msgr2=0x7f460c149ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:35.210 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:35.232+0000 7f4611484640 1 -- 172.21.15.17:0/3706602791 shutdown_connections 2024-03-29T11:20:35.210 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:35.232+0000 7f4611484640 1 -- 172.21.15.17:0/3706602791 wait complete. 2024-03-29T11:20:35.210 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:35.232+0000 7f4611484640 1 librados: shutdown 2024-03-29T11:20:35.228 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:42.035 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:42.187 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.210+0000 7f9ad12e4640 10 monclient: get_monmap_and_config 2024-03-29T11:20:42.187 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.210+0000 7f9ad12e4640 10 monclient: build_initial_monmap 2024-03-29T11:20:42.187 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.210+0000 7f9ad12e4640 10 monclient: monmap: 2024-03-29T11:20:42.187 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:20:42.187 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:20:42.188 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:20:42.211263+0000 2024-03-29T11:20:42.188 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:20:42.211263+0000 2024-03-29T11:20:42.188 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:42.188 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:42.188 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:20:42.188 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:20:42.188 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:20:42.188 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:42.188 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.211+0000 7f9ad12e4640 1 Processor -- start 2024-03-29T11:20:42.188 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.211+0000 7f9ad12e4640 1 -- start start 2024-03-29T11:20:42.188 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.211+0000 7f9ad12e4640 10 monclient: init 2024-03-29T11:20:42.189 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.211+0000 7f9ad12e4640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:42.189 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.211+0000 7f9ad12e4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:20:42.189 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.211+0000 7f9ad12e4640 10 monclient(hunting): picked mon.noname-b con 0x7f9acc1457f0 addr v1:172.21.15.17:6790/0 2024-03-29T11:20:42.189 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.211+0000 7f9ad12e4640 10 monclient(hunting): picked mon.noname-c con 0x7f9acc13f650 addr v1:172.21.15.94:6789/0 2024-03-29T11:20:42.189 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.212+0000 7f9ad12e4640 10 monclient(hunting): picked mon.noname-a con 0x7f9acc0a9d70 addr v1:172.21.15.17:6789/0 2024-03-29T11:20:42.189 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.212+0000 7f9ad12e4640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9acc0aaef0 con 0x7f9acc0a9d70 2024-03-29T11:20:42.189 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.212+0000 7f9ad12e4640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9acc145bd0 con 0x7f9acc13f650 2024-03-29T11:20:42.190 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.212+0000 7f9ad12e4640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9acc145eb0 con 0x7f9acc1457f0 2024-03-29T11:20:42.190 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.212+0000 7f9ad12e4640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:42.190 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.212+0000 7f9ad12e4640 10 monclient(hunting): authenticate will time out at 948.416922s 2024-03-29T11:20:42.198 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.218+0000 7f9acb7fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f9acc0a9d70 0x7f9acc14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:50949/0 (socket says 172.21.15.17:50949) 2024-03-29T11:20:42.198 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.218+0000 7f9acb7fe640 1 -- 172.21.15.17:0/3432357106 learned_addr learned my addr 172.21.15.17:0/3432357106 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:20:42.198 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.219+0000 7f9aca7fc640 1 -- 172.21.15.17:0/3432357106 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2500241786 0 0) 0x7f9acc0aaef0 con 0x7f9acc0a9d70 2024-03-29T11:20:42.198 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.219+0000 7f9aca7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:42.198 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.219+0000 7f9aca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:42.198 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.219+0000 7f9aca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:42.199 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.219+0000 7f9aca7fc640 10 monclient(hunting): my global_id is 4284 2024-03-29T11:20:42.199 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.219+0000 7f9aca7fc640 1 -- 172.21.15.17:0/3432357106 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ab8003520 con 0x7f9acc0a9d70 2024-03-29T11:20:42.199 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.219+0000 7f9aca7fc640 1 -- 172.21.15.17:0/3432357106 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4027812888 0 0) 0x7f9ab8003520 con 0x7f9acc0a9d70 2024-03-29T11:20:42.199 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.219+0000 7f9aca7fc640 1 -- 172.21.15.17:0/3432357106 >> v1:172.21.15.17:6790/0 conn(0x7f9acc1457f0 legacy=0x7f9acc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:42.199 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.219+0000 7f9aca7fc640 1 -- 172.21.15.17:0/3432357106 >> v1:172.21.15.94:6789/0 conn(0x7f9acc13f650 legacy=0x7f9acc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:20:42.199 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.219+0000 7f9aca7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:20:42.199 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.219+0000 7f9aca7fc640 1 monclient: found mon.noname-a 2024-03-29T11:20:42.199 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.219+0000 7f9aca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:42.199 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.219+0000 7f9aca7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.17:6789/0 2024-03-29T11:20:42.199 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.219+0000 7f9aca7fc640 1 -- 172.21.15.17:0/3432357106 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9acc146340 con 0x7f9acc0a9d70 2024-03-29T11:20:42.199 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.219+0000 7f9aca7fc640 10 monclient: _finish_auth 0 2024-03-29T11:20:42.200 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.219+0000 7f9aca7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:20:42.200 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.219+0000 7f9aca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:42.200 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.219+0000 7f9aca7fc640 1 -- 172.21.15.17:0/3432357106 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f9ac4001a10 con 0x7f9acc0a9d70 2024-03-29T11:20:42.200 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.219+0000 7f9aca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:42.200 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.219+0000 7f9aca7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:42.200 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.219+0000 7f9aca7fc640 10 monclient: dump: 2024-03-29T11:20:42.200 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:20:42.200 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:20:42.200 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:42.200 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:42.200 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:42.200 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:42.200 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:20:42.201 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:20:42.201 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:20:42.201 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:42.201 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.219+0000 7f9ad12e4640 5 monclient: authenticate success, global_id 4284 2024-03-29T11:20:42.201 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.219+0000 7f9ad12e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:42.201 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.219+0000 7f9aca7fc640 1 -- 172.21.15.17:0/3432357106 <== mon.0 v1:172.21.15.17:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ac4001ca0 con 0x7f9acc0a9d70 2024-03-29T11:20:42.201 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.219+0000 7f9aca7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:42.201 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.219+0000 7f9ad12e4640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:42.201 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.219+0000 7f9ad12e4640 4 set_mon_vals no callback set 2024-03-29T11:20:42.201 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.220+0000 7f9ad12e4640 10 monclient: shutdown 2024-03-29T11:20:42.201 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.220+0000 7f9ad12e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:42.201 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.220+0000 7f9ad12e4640 1 -- 172.21.15.17:0/3432357106 >> v1:172.21.15.17:6789/0 conn(0x7f9acc0a9d70 legacy=0x7f9acc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:42.203 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.223+0000 7f9ad12e4640 1 -- 172.21.15.17:0/3432357106 shutdown_connections 2024-03-29T11:20:42.203 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.223+0000 7f9ad12e4640 1 -- 172.21.15.17:0/3432357106 >> 172.21.15.17:0/3432357106 conn(0x7f9acc066a00 msgr2=0x7f9acc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:42.203 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.224+0000 7f9ad12e4640 1 -- 172.21.15.17:0/3432357106 shutdown_connections 2024-03-29T11:20:42.203 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.224+0000 7f9ad12e4640 1 -- 172.21.15.17:0/3432357106 wait complete. 2024-03-29T11:20:42.203 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.225+0000 7f9ad12e4640 10 monclient: build_initial_monmap 2024-03-29T11:20:42.203 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.225+0000 7f9ad12e4640 10 monclient: monmap: 2024-03-29T11:20:42.203 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:20:42.203 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:42.203 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:20:42.203 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:20:42.204 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:42.204 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:42.204 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:20:42.204 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:20:42.204 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:20:42.204 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:42.204 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.225+0000 7f9ad12e4640 1 librados: starting msgr at 2024-03-29T11:20:42.204 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.225+0000 7f9ad12e4640 1 librados: starting objecter 2024-03-29T11:20:42.204 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.225+0000 7f9ad12e4640 1 Processor -- start 2024-03-29T11:20:42.206 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.226+0000 7f9ad12e4640 1 -- start start 2024-03-29T11:20:42.206 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.226+0000 7f9ad12e4640 1 librados: setting wanted keys 2024-03-29T11:20:42.206 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.226+0000 7f9ad12e4640 1 librados: calling monclient init 2024-03-29T11:20:42.210 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.226+0000 7f9ad12e4640 10 monclient: init 2024-03-29T11:20:42.210 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.227+0000 7f9ad12e4640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:42.210 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.227+0000 7f9ad12e4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:20:42.210 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.227+0000 7f9ad12e4640 10 monclient(hunting): picked mon.noname-a con 0x7f9acc0a9d70 addr v1:172.21.15.17:6789/0 2024-03-29T11:20:42.210 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.227+0000 7f9ad12e4640 10 monclient(hunting): picked mon.noname-c con 0x7f9acc13f650 addr v1:172.21.15.17:6790/0 2024-03-29T11:20:42.211 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.227+0000 7f9ad12e4640 10 monclient(hunting): picked mon.noname-b con 0x7f9acc1457f0 addr v1:172.21.15.94:6789/0 2024-03-29T11:20:42.211 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.227+0000 7f9ad12e4640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9acc1e74d0 con 0x7f9acc0a9d70 2024-03-29T11:20:42.211 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.227+0000 7f9ad12e4640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9acc13fda0 con 0x7f9acc1457f0 2024-03-29T11:20:42.211 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.227+0000 7f9ad12e4640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9acc140080 con 0x7f9acc13f650 2024-03-29T11:20:42.211 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.227+0000 7f9ad12e4640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:42.211 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.227+0000 7f9ad12e4640 10 monclient(hunting): authenticate will time out at 948.432070s 2024-03-29T11:20:42.211 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.231+0000 7f9acaffd640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f9acc0a9d70 0x7f9acc1e7990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:51089/0 (socket says 172.21.15.17:51089) 2024-03-29T11:20:42.212 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.231+0000 7f9acaffd640 1 -- 172.21.15.17:0/4080815828 learned_addr learned my addr 172.21.15.17:0/4080815828 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:20:42.212 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.232+0000 7f9ac3fff640 10 client.?.objecter ms_handle_connect 0x7f9acc13f650 2024-03-29T11:20:42.212 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.232+0000 7f9ac3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:42.212 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.232+0000 7f9ac3fff640 1 -- 172.21.15.17:0/4080815828 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3792391001 0 0) 0x7f9acc140080 con 0x7f9acc13f650 2024-03-29T11:20:42.212 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.232+0000 7f9ac3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:42.212 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.232+0000 7f9ac3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:42.212 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.232+0000 7f9ac3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:42.212 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.232+0000 7f9ac3fff640 10 monclient(hunting): my global_id is 4298 2024-03-29T11:20:42.212 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.232+0000 7f9ac3fff640 1 -- 172.21.15.17:0/4080815828 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9aac004eb0 con 0x7f9acc13f650 2024-03-29T11:20:42.212 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.233+0000 7f9ac3fff640 1 -- 172.21.15.17:0/4080815828 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3166420978 0 0) 0x7f9aac004eb0 con 0x7f9acc13f650 2024-03-29T11:20:42.212 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.233+0000 7f9ac3fff640 1 -- 172.21.15.17:0/4080815828 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9aac005300 con 0x7f9acc13f650 2024-03-29T11:20:42.213 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.233+0000 7f9ac3fff640 1 -- 172.21.15.17:0/4080815828 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f9ab4001510 con 0x7f9acc13f650 2024-03-29T11:20:42.213 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.233+0000 7f9ac3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:42.213 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.234+0000 7f9ac3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:42.213 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.234+0000 7f9ac3fff640 10 monclient(hunting): dump: 2024-03-29T11:20:42.213 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:20:42.213 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:20:42.213 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:42.213 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:42.213 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:42.213 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:42.213 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:20:42.213 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:20:42.213 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:20:42.214 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:42.214 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.234+0000 7f9ac3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:42.214 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.234+0000 7f9ac3fff640 10 client.?.objecter ms_handle_connect 0x7f9acc0a9d70 2024-03-29T11:20:42.214 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.234+0000 7f9ac3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:42.214 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.234+0000 7f9ac3fff640 1 -- 172.21.15.17:0/4080815828 <== mon.2 v1:172.21.15.17:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3759048028 0 0) 0x7f9aac005300 con 0x7f9acc13f650 2024-03-29T11:20:42.214 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.234+0000 7f9ac3fff640 1 -- 172.21.15.17:0/4080815828 >> v1:172.21.15.94:6789/0 conn(0x7f9acc1457f0 legacy=0x7f9acc1ef7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:42.214 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.234+0000 7f9ac3fff640 1 -- 172.21.15.17:0/4080815828 >> v1:172.21.15.17:6789/0 conn(0x7f9acc0a9d70 legacy=0x7f9acc1e7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:42.214 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.234+0000 7f9ac3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:20:42.214 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.234+0000 7f9ac3fff640 1 monclient: found mon.c 2024-03-29T11:20:42.214 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.234+0000 7f9ac3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:42.214 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.234+0000 7f9ac3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:20:42.214 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.234+0000 7f9ac3fff640 1 -- 172.21.15.17:0/4080815828 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9acc1e80a0 con 0x7f9acc13f650 2024-03-29T11:20:42.214 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.234+0000 7f9ac3fff640 10 monclient: _finish_auth 0 2024-03-29T11:20:42.215 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.234+0000 7f9ac3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:20:42.215 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.234+0000 7f9ac3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:42.215 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.234+0000 7f9ad12e4640 5 monclient: authenticate success, global_id 4298 2024-03-29T11:20:42.215 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.234+0000 7f9ad12e4640 10 monclient: _renew_subs 2024-03-29T11:20:42.215 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.234+0000 7f9ad12e4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:20:42.215 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.234+0000 7f9ad12e4640 1 -- 172.21.15.17:0/4080815828 --> v1:172.21.15.17:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9acc1e8350 con 0x7f9acc13f650 2024-03-29T11:20:42.215 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.234+0000 7f9ad12e4640 10 client.4298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:42.215 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.234+0000 7f9ad12e4640 10 monclient: _renew_subs 2024-03-29T11:20:42.215 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.234+0000 7f9ad12e4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:20:42.215 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.234+0000 7f9ad12e4640 1 -- 172.21.15.17:0/4080815828 --> v1:172.21.15.17:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9acc142ca0 con 0x7f9acc13f650 2024-03-29T11:20:42.215 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.234+0000 7f9ad12e4640 1 librados: init done 2024-03-29T11:20:42.217 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.235+0000 7f9ac3fff640 1 -- 172.21.15.17:0/4080815828 <== mon.2 v1:172.21.15.17:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ab4001ae0 con 0x7f9acc13f650 2024-03-29T11:20:42.217 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.235+0000 7f9ac3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:42.218 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.235+0000 7f9ac9ffb640 4 set_mon_vals no callback set 2024-03-29T11:20:42.218 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.236+0000 7f9ac3fff640 1 -- 172.21.15.17:0/4080815828 <== mon.2 v1:172.21.15.17:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f9ab4001510 con 0x7f9acc13f650 2024-03-29T11:20:42.218 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.236+0000 7f9ac3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:42.218 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.236+0000 7f9ac3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:42.218 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.236+0000 7f9ac3fff640 10 monclient: dump: 2024-03-29T11:20:42.218 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:20:42.218 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:20:42.218 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:42.218 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:42.219 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:42.219 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:42.219 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:20:42.219 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:20:42.219 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:20:42.219 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:42.219 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.238+0000 7f9ac3fff640 1 -- 172.21.15.17:0/4080815828 <== mon.2 v1:172.21.15.17:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f9ab4019780 con 0x7f9acc13f650 2024-03-29T11:20:42.219 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.238+0000 7f9ad12e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:42.219 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.238+0000 7f9ad12e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:42.219 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.238+0000 7f9ad12e4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:20:42.219 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.238+0000 7f9ad12e4640 1 -- 172.21.15.17:0/4080815828 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ab0005080 con 0x7f9acc13f650 2024-03-29T11:20:42.219 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.239+0000 7f9acaffd640 10 monclient: get_auth_request con 0x7f9aac06e1d0 auth_method 0 2024-03-29T11:20:42.219 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.239+0000 7f9ac3fff640 1 -- 172.21.15.17:0/4080815828 <== mon.2 v1:172.21.15.17:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f9ab4053140 con 0x7f9acc13f650 2024-03-29T11:20:42.219 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.239+0000 7f9ac3fff640 10 client.4298.objecter ms_dispatch 0x7f9acc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:42.220 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.239+0000 7f9ac3fff640 3 client.4298.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:42.220 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.239+0000 7f9ac3fff640 3 client.4298.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:42.220 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.239+0000 7f9ac3fff640 20 client.4298.objecter dump_active .. 0 homeless 2024-03-29T11:20:42.220 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.239+0000 7f9ac3fff640 10 client.4298.objecter ms_handle_connect 0x7f9aac06e1d0 2024-03-29T11:20:42.224 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.247+0000 7f9ac3fff640 1 -- 172.21.15.17:0/4080815828 <== mon.2 v1:172.21.15.17:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9ab404f3e0 con 0x7f9acc13f650 2024-03-29T11:20:42.224 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.248+0000 7f9ac3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:42.225 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.248+0000 7f9ac3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:42.555 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.578+0000 7f9ad12e4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:42.555 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.578+0000 7f9ad12e4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:42.555 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.578+0000 7f9ad12e4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:20:42.555 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.578+0000 7f9ad12e4640 1 -- 172.21.15.17:0/4080815828 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ab0005330 con 0x7f9acc13f650 2024-03-29T11:20:42.556 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.579+0000 7f9ac3fff640 1 -- 172.21.15.17:0/4080815828 <== mon.2 v1:172.21.15.17:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f9ab404f3e0 con 0x7f9acc13f650 2024-03-29T11:20:42.556 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.579+0000 7f9ac3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:42.556 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.579+0000 7f9ac3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:42.556 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:20:42.557 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:42.559 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.583+0000 7f9ad12e4640 10 librados: watch_flush enter 2024-03-29T11:20:42.559 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.583+0000 7f9ad12e4640 10 librados: watch_flush exit 2024-03-29T11:20:42.559 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.583+0000 7f9ad12e4640 20 client.4298.objecter shutdown clearing up homeless session... 2024-03-29T11:20:42.560 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.583+0000 7f9ad12e4640 10 client.4298.objecter successfully canceled tick 2024-03-29T11:20:42.560 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.583+0000 7f9ad12e4640 1 -- 172.21.15.17:0/4080815828 >> v1:172.21.15.17:6816/3236185052 conn(0x7f9aac06e1d0 legacy=0x7f9aac070690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:42.560 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.583+0000 7f9ad12e4640 10 monclient: shutdown 2024-03-29T11:20:42.560 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.583+0000 7f9ad12e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:42.560 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.583+0000 7f9ad12e4640 1 -- 172.21.15.17:0/4080815828 >> v1:172.21.15.17:6790/0 conn(0x7f9acc13f650 legacy=0x7f9acc1423b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:42.560 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.583+0000 7f9ad12e4640 1 -- 172.21.15.17:0/4080815828 shutdown_connections 2024-03-29T11:20:42.560 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.583+0000 7f9ad12e4640 1 -- 172.21.15.17:0/4080815828 >> 172.21.15.17:0/4080815828 conn(0x7f9acc066a00 msgr2=0x7f9acc1d2590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:42.563 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.586+0000 7f9ad12e4640 1 -- 172.21.15.17:0/4080815828 shutdown_connections 2024-03-29T11:20:42.563 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.586+0000 7f9ad12e4640 1 -- 172.21.15.17:0/4080815828 wait complete. 2024-03-29T11:20:42.563 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:42.586+0000 7f9ad12e4640 1 librados: shutdown 2024-03-29T11:20:42.580 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:49.387 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:49.537 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.560+0000 7f4bff286640 10 monclient: get_monmap_and_config 2024-03-29T11:20:49.537 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.560+0000 7f4bff286640 10 monclient: build_initial_monmap 2024-03-29T11:20:49.537 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.560+0000 7f4bff286640 10 monclient: monmap: 2024-03-29T11:20:49.538 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:20:49.538 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:20:49.538 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:20:49.561046+0000 2024-03-29T11:20:49.538 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:20:49.561046+0000 2024-03-29T11:20:49.538 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:49.538 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:49.538 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:20:49.538 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:20:49.538 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:20:49.538 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:49.539 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.562+0000 7f4bff286640 1 Processor -- start 2024-03-29T11:20:49.539 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.562+0000 7f4bff286640 1 -- start start 2024-03-29T11:20:49.539 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.562+0000 7f4bff286640 10 monclient: init 2024-03-29T11:20:49.539 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.563+0000 7f4bff286640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:49.540 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.563+0000 7f4bff286640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:20:49.540 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.563+0000 7f4bff286640 10 monclient(hunting): picked mon.noname-a con 0x7f4bf80b4100 addr v1:172.21.15.17:6789/0 2024-03-29T11:20:49.540 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.563+0000 7f4bff286640 10 monclient(hunting): picked mon.noname-b con 0x7f4bf813ee90 addr v1:172.21.15.17:6790/0 2024-03-29T11:20:49.540 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.563+0000 7f4bff286640 10 monclient(hunting): picked mon.noname-c con 0x7f4bf80a9ac0 addr v1:172.21.15.94:6789/0 2024-03-29T11:20:49.540 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.563+0000 7f4bff286640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bf80af6f0 con 0x7f4bf80b4100 2024-03-29T11:20:49.540 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.563+0000 7f4bff286640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bf80aac80 con 0x7f4bf80a9ac0 2024-03-29T11:20:49.540 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.563+0000 7f4bff286640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bf80aaf60 con 0x7f4bf813ee90 2024-03-29T11:20:49.540 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.563+0000 7f4bff286640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:49.540 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.563+0000 7f4bff286640 10 monclient(hunting): authenticate will time out at 955.768323s 2024-03-29T11:20:49.544 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.566+0000 7f4bfd022640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f4bf80b4100 0x7f4bf80adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:42657/0 (socket says 172.21.15.17:42657) 2024-03-29T11:20:49.544 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.566+0000 7f4bfd022640 1 -- 172.21.15.17:0/1801026714 learned_addr learned my addr 172.21.15.17:0/1801026714 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:20:49.546 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.569+0000 7f4beffff640 1 -- 172.21.15.17:0/1801026714 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2013606166 0 0) 0x7f4bf80af6f0 con 0x7f4bf80b4100 2024-03-29T11:20:49.546 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.569+0000 7f4beffff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:49.546 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.569+0000 7f4beffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:49.546 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.569+0000 7f4beffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:49.546 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.569+0000 7f4beffff640 10 monclient(hunting): my global_id is 4290 2024-03-29T11:20:49.546 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.569+0000 7f4beffff640 1 -- 172.21.15.17:0/1801026714 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4be4003520 con 0x7f4bf80b4100 2024-03-29T11:20:49.547 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.570+0000 7f4beffff640 1 -- 172.21.15.17:0/1801026714 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2169271216 0 0) 0x7f4bf80aaf60 con 0x7f4bf813ee90 2024-03-29T11:20:49.547 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.570+0000 7f4beffff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:49.547 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.570+0000 7f4beffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:49.547 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.570+0000 7f4beffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:49.547 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.570+0000 7f4beffff640 10 monclient(hunting): my global_id is 4301 2024-03-29T11:20:49.547 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.570+0000 7f4beffff640 1 -- 172.21.15.17:0/1801026714 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bf80af6f0 con 0x7f4bf813ee90 2024-03-29T11:20:49.547 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.570+0000 7f4beffff640 1 -- 172.21.15.17:0/1801026714 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4086191026 0 0) 0x7f4be4003520 con 0x7f4bf80b4100 2024-03-29T11:20:49.548 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.570+0000 7f4beffff640 1 -- 172.21.15.17:0/1801026714 >> v1:172.21.15.17:6790/0 conn(0x7f4bf813ee90 legacy=0x7f4bf813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:49.548 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.570+0000 7f4beffff640 1 -- 172.21.15.17:0/1801026714 >> v1:172.21.15.94:6789/0 conn(0x7f4bf80a9ac0 legacy=0x7f4bf8142ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:20:49.548 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.570+0000 7f4beffff640 10 monclient: _finish_hunting 0 2024-03-29T11:20:49.548 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.570+0000 7f4beffff640 1 monclient: found mon.noname-a 2024-03-29T11:20:49.548 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.570+0000 7f4beffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:49.548 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.570+0000 7f4beffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.17:6789/0 2024-03-29T11:20:49.548 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.570+0000 7f4beffff640 1 -- 172.21.15.17:0/1801026714 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bf813e970 con 0x7f4bf80b4100 2024-03-29T11:20:49.548 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.570+0000 7f4beffff640 10 monclient: _finish_auth 0 2024-03-29T11:20:49.548 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.570+0000 7f4beffff640 10 monclient: _check_auth_tickets 2024-03-29T11:20:49.548 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.570+0000 7f4beffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:49.548 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.570+0000 7f4beffff640 1 -- 172.21.15.17:0/1801026714 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f4be8001690 con 0x7f4bf80b4100 2024-03-29T11:20:49.549 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.570+0000 7f4beffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:49.549 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.570+0000 7f4beffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:49.549 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.570+0000 7f4beffff640 10 monclient: dump: 2024-03-29T11:20:49.549 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:20:49.549 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:20:49.549 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:49.549 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:49.549 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:49.549 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:49.549 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:20:49.549 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:20:49.549 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:20:49.549 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:49.550 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.570+0000 7f4beffff640 1 -- 172.21.15.17:0/1801026714 <== mon.0 v1:172.21.15.17:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4be80017f0 con 0x7f4bf80b4100 2024-03-29T11:20:49.550 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.570+0000 7f4beffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:49.550 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.570+0000 7f4beffff640 1 -- 172.21.15.17:0/1801026714 <== mon.0 v1:172.21.15.17:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f4be8001c00 con 0x7f4bf80b4100 2024-03-29T11:20:49.550 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.570+0000 7f4beffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:49.550 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.570+0000 7f4beffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:49.550 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.570+0000 7f4beffff640 10 monclient: dump: 2024-03-29T11:20:49.550 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:20:49.550 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:20:49.550 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:49.550 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:49.551 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:49.551 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:49.551 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:20:49.551 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:20:49.551 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:20:49.551 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:49.552 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.571+0000 7f4bff286640 5 monclient: authenticate success, global_id 4290 2024-03-29T11:20:49.552 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.571+0000 7f4bff286640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:49.553 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.571+0000 7f4bff286640 4 set_mon_vals no callback set 2024-03-29T11:20:49.553 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.572+0000 7f4bff286640 10 monclient: shutdown 2024-03-29T11:20:49.553 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.572+0000 7f4bff286640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:49.553 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.572+0000 7f4bff286640 1 -- 172.21.15.17:0/1801026714 >> v1:172.21.15.17:6789/0 conn(0x7f4bf80b4100 legacy=0x7f4bf80adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:49.553 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.572+0000 7f4bff286640 1 -- 172.21.15.17:0/1801026714 shutdown_connections 2024-03-29T11:20:49.553 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.572+0000 7f4bff286640 1 -- 172.21.15.17:0/1801026714 >> 172.21.15.17:0/1801026714 conn(0x7f4bf80b35e0 msgr2=0x7f4bf80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:49.553 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.572+0000 7f4bff286640 1 -- 172.21.15.17:0/1801026714 shutdown_connections 2024-03-29T11:20:49.553 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.573+0000 7f4bff286640 1 -- 172.21.15.17:0/1801026714 wait complete. 2024-03-29T11:20:49.553 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.573+0000 7f4bff286640 10 monclient: build_initial_monmap 2024-03-29T11:20:49.553 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.573+0000 7f4bff286640 10 monclient: monmap: 2024-03-29T11:20:49.553 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:20:49.553 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:49.553 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:20:49.554 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:20:49.554 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:49.554 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:49.554 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:20:49.554 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:20:49.554 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:20:49.554 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:49.554 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.574+0000 7f4bff286640 1 librados: starting msgr at 2024-03-29T11:20:49.554 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.574+0000 7f4bff286640 1 librados: starting objecter 2024-03-29T11:20:49.554 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.574+0000 7f4bff286640 1 Processor -- start 2024-03-29T11:20:49.554 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.574+0000 7f4bff286640 1 -- start start 2024-03-29T11:20:49.554 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.574+0000 7f4bff286640 1 librados: setting wanted keys 2024-03-29T11:20:49.554 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.574+0000 7f4bff286640 1 librados: calling monclient init 2024-03-29T11:20:49.554 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.574+0000 7f4bff286640 10 monclient: init 2024-03-29T11:20:49.556 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.576+0000 7f4bff286640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:49.556 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.576+0000 7f4bff286640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:49.556 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.576+0000 7f4bff286640 10 monclient(hunting): picked mon.noname-c con 0x7f4bf80a9ac0 addr v1:172.21.15.17:6790/0 2024-03-29T11:20:49.556 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.576+0000 7f4bff286640 10 monclient(hunting): picked mon.noname-a con 0x7f4bf80b4100 addr v1:172.21.15.17:6789/0 2024-03-29T11:20:49.556 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.576+0000 7f4bff286640 10 monclient(hunting): picked mon.noname-b con 0x7f4bf813ee90 addr v1:172.21.15.94:6789/0 2024-03-29T11:20:49.556 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.576+0000 7f4bff286640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bf8137190 con 0x7f4bf80b4100 2024-03-29T11:20:49.556 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.576+0000 7f4bff286640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bf81411b0 con 0x7f4bf813ee90 2024-03-29T11:20:49.556 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.576+0000 7f4bff286640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bf8141490 con 0x7f4bf80a9ac0 2024-03-29T11:20:49.556 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.576+0000 7f4bff286640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:49.556 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.576+0000 7f4bff286640 10 monclient(hunting): authenticate will time out at 955.781238s 2024-03-29T11:20:49.560 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.582+0000 7f4bfc821640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f4bf80b4100 0x7f4bf8136370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:42775/0 (socket says 172.21.15.17:42775) 2024-03-29T11:20:49.560 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.582+0000 7f4bfc821640 1 -- 172.21.15.17:0/4252447188 learned_addr learned my addr 172.21.15.17:0/4252447188 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:20:49.560 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.582+0000 7f4bedffb640 10 client.?.objecter ms_handle_connect 0x7f4bf80b4100 2024-03-29T11:20:49.560 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.582+0000 7f4bedffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:49.560 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.582+0000 7f4bedffb640 1 -- 172.21.15.17:0/4252447188 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 160157618 0 0) 0x7f4bf8137190 con 0x7f4bf80b4100 2024-03-29T11:20:49.561 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.582+0000 7f4bedffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:49.561 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.582+0000 7f4bedffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:49.561 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.582+0000 7f4bedffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:49.561 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.582+0000 7f4bedffb640 10 monclient(hunting): my global_id is 4293 2024-03-29T11:20:49.561 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.582+0000 7f4bedffb640 1 -- 172.21.15.17:0/4252447188 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bdc004eb0 con 0x7f4bf80b4100 2024-03-29T11:20:49.561 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.583+0000 7f4bedffb640 1 -- 172.21.15.17:0/4252447188 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1293063621 0 0) 0x7f4bdc004eb0 con 0x7f4bf80b4100 2024-03-29T11:20:49.561 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.583+0000 7f4bedffb640 1 -- 172.21.15.17:0/4252447188 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4bdc005300 con 0x7f4bf80b4100 2024-03-29T11:20:49.561 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.583+0000 7f4bedffb640 10 client.?.objecter ms_handle_connect 0x7f4bf813ee90 2024-03-29T11:20:49.561 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.583+0000 7f4bedffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:49.561 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.583+0000 7f4bedffb640 1 -- 172.21.15.17:0/4252447188 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f4bf4001a20 con 0x7f4bf80b4100 2024-03-29T11:20:49.561 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.583+0000 7f4bedffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:49.561 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.583+0000 7f4bedffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:49.561 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.583+0000 7f4bedffb640 10 monclient(hunting): dump: 2024-03-29T11:20:49.562 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:20:49.562 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:20:49.562 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:49.562 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:49.562 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:49.562 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:49.562 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:20:49.562 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:20:49.562 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:20:49.562 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:49.562 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.583+0000 7f4bedffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:49.562 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.583+0000 7f4bedffb640 1 -- 172.21.15.17:0/4252447188 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3416208153 0 0) 0x7f4bf81411b0 con 0x7f4bf813ee90 2024-03-29T11:20:49.562 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.583+0000 7f4bedffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:49.562 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.583+0000 7f4bedffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:49.563 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.583+0000 7f4bedffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:49.563 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.583+0000 7f4bedffb640 10 monclient(hunting): my global_id is 4306 2024-03-29T11:20:49.563 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.583+0000 7f4bedffb640 1 -- 172.21.15.17:0/4252447188 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bdc0058a0 con 0x7f4bf813ee90 2024-03-29T11:20:49.563 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.586+0000 7f4bedffb640 1 -- 172.21.15.17:0/4252447188 <== mon.0 v1:172.21.15.17:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2026169726 0 0) 0x7f4bdc005300 con 0x7f4bf80b4100 2024-03-29T11:20:49.563 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.586+0000 7f4bedffb640 1 -- 172.21.15.17:0/4252447188 >> v1:172.21.15.17:6790/0 conn(0x7f4bf80a9ac0 legacy=0x7f4bf8137d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:49.563 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.586+0000 7f4bedffb640 1 -- 172.21.15.17:0/4252447188 >> v1:172.21.15.94:6789/0 conn(0x7f4bf813ee90 legacy=0x7f4bf8136a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:49.563 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.587+0000 7f4bedffb640 10 monclient: _finish_hunting 0 2024-03-29T11:20:49.563 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.587+0000 7f4bedffb640 1 monclient: found mon.a 2024-03-29T11:20:49.564 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.587+0000 7f4bedffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:49.564 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.587+0000 7f4bedffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:20:49.564 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.587+0000 7f4bedffb640 1 -- 172.21.15.17:0/4252447188 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bf8141770 con 0x7f4bf80b4100 2024-03-29T11:20:49.564 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.587+0000 7f4bedffb640 10 monclient: _finish_auth 0 2024-03-29T11:20:49.564 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.587+0000 7f4bedffb640 10 monclient: _check_auth_tickets 2024-03-29T11:20:49.564 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.587+0000 7f4bedffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:49.564 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.587+0000 7f4bff286640 5 monclient: authenticate success, global_id 4293 2024-03-29T11:20:49.564 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.587+0000 7f4bff286640 10 monclient: _renew_subs 2024-03-29T11:20:49.564 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.587+0000 7f4bff286640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:20:49.564 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.587+0000 7f4bff286640 1 -- 172.21.15.17:0/4252447188 --> v1:172.21.15.17:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bf8141a50 con 0x7f4bf80b4100 2024-03-29T11:20:49.564 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.587+0000 7f4bff286640 10 client.4293.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:49.565 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.587+0000 7f4bff286640 10 monclient: _renew_subs 2024-03-29T11:20:49.565 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.587+0000 7f4bff286640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:20:49.565 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.587+0000 7f4bff286640 1 -- 172.21.15.17:0/4252447188 --> v1:172.21.15.17:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bf81421c0 con 0x7f4bf80b4100 2024-03-29T11:20:49.565 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.587+0000 7f4bff286640 1 librados: init done 2024-03-29T11:20:49.568 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.589+0000 7f4bedffb640 1 -- 172.21.15.17:0/4252447188 <== mon.0 v1:172.21.15.17:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4bf4001cc0 con 0x7f4bf80b4100 2024-03-29T11:20:49.568 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.589+0000 7f4bedffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:49.568 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.589+0000 7f4bedffb640 1 -- 172.21.15.17:0/4252447188 <== mon.0 v1:172.21.15.17:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f4bf4001a20 con 0x7f4bf80b4100 2024-03-29T11:20:49.568 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.589+0000 7f4bedffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:49.568 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.589+0000 7f4bedffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:49.568 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.589+0000 7f4bedffb640 10 monclient: dump: 2024-03-29T11:20:49.568 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:20:49.568 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:20:49.568 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:49.569 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:49.569 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:49.569 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:49.569 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:20:49.569 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:20:49.569 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:20:49.569 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:49.569 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.589+0000 7f4bff286640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:49.569 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.589+0000 7f4bff286640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:49.569 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.589+0000 7f4bff286640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:20:49.569 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.589+0000 7f4bff286640 1 -- 172.21.15.17:0/4252447188 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bf813f5c0 con 0x7f4bf80b4100 2024-03-29T11:20:49.569 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.591+0000 7f4beeffd640 4 set_mon_vals no callback set 2024-03-29T11:20:49.570 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.593+0000 7f4bedffb640 1 -- 172.21.15.17:0/4252447188 <== mon.0 v1:172.21.15.17:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f4bf4003940 con 0x7f4bf80b4100 2024-03-29T11:20:49.574 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.594+0000 7f4bedffb640 1 -- 172.21.15.17:0/4252447188 <== mon.0 v1:172.21.15.17:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f4bf40848b0 con 0x7f4bf80b4100 2024-03-29T11:20:49.575 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.594+0000 7f4bedffb640 10 client.4293.objecter ms_dispatch 0x7f4bf8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:49.575 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.594+0000 7f4bedffb640 3 client.4293.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:49.575 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.594+0000 7f4bedffb640 3 client.4293.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:49.575 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.594+0000 7f4bedffb640 20 client.4293.objecter dump_active .. 0 homeless 2024-03-29T11:20:49.575 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.595+0000 7f4bfd022640 10 monclient: get_auth_request con 0x7f4bdc06e400 auth_method 0 2024-03-29T11:20:49.575 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.595+0000 7f4bedffb640 10 client.4293.objecter ms_handle_connect 0x7f4bdc06e400 2024-03-29T11:20:49.579 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.602+0000 7f4bedffb640 1 -- 172.21.15.17:0/4252447188 <== mon.0 v1:172.21.15.17:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4bf813f5c0 con 0x7f4bf80b4100 2024-03-29T11:20:49.579 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.602+0000 7f4bedffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:49.579 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.602+0000 7f4bedffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:49.911 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.934+0000 7f4bff286640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:49.911 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.934+0000 7f4bff286640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:49.911 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.934+0000 7f4bff286640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:20:49.911 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.934+0000 7f4bff286640 1 -- 172.21.15.17:0/4252447188 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4bf813f9a0 con 0x7f4bf80b4100 2024-03-29T11:20:49.912 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.935+0000 7f4bedffb640 1 -- 172.21.15.17:0/4252447188 <== mon.0 v1:172.21.15.17:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f4bf813f9a0 con 0x7f4bf80b4100 2024-03-29T11:20:49.912 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.935+0000 7f4bedffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:49.912 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.935+0000 7f4bedffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:49.912 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:20:49.913 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:49.916 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.939+0000 7f4bff286640 10 librados: watch_flush enter 2024-03-29T11:20:49.916 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.939+0000 7f4bff286640 10 librados: watch_flush exit 2024-03-29T11:20:49.916 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.939+0000 7f4bff286640 20 client.4293.objecter shutdown clearing up homeless session... 2024-03-29T11:20:49.916 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.939+0000 7f4bff286640 10 client.4293.objecter successfully canceled tick 2024-03-29T11:20:49.916 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.939+0000 7f4bff286640 1 -- 172.21.15.17:0/4252447188 >> v1:172.21.15.17:6816/3236185052 conn(0x7f4bdc06e400 legacy=0x7f4bdc0708c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:49.916 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.939+0000 7f4bff286640 10 monclient: shutdown 2024-03-29T11:20:49.916 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.939+0000 7f4bff286640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:49.916 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.939+0000 7f4bff286640 1 -- 172.21.15.17:0/4252447188 >> v1:172.21.15.17:6789/0 conn(0x7f4bf80b4100 legacy=0x7f4bf8136370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:49.916 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.939+0000 7f4bff286640 1 -- 172.21.15.17:0/4252447188 shutdown_connections 2024-03-29T11:20:49.917 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.939+0000 7f4bff286640 1 -- 172.21.15.17:0/4252447188 >> 172.21.15.17:0/4252447188 conn(0x7f4bf80b35e0 msgr2=0x7f4bf80adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:49.918 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.942+0000 7f4bff286640 1 -- 172.21.15.17:0/4252447188 shutdown_connections 2024-03-29T11:20:49.918 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.942+0000 7f4bff286640 1 -- 172.21.15.17:0/4252447188 wait complete. 2024-03-29T11:20:49.918 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:49.942+0000 7f4bff286640 1 librados: shutdown 2024-03-29T11:20:49.934 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:56.742 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:56.895 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.918+0000 7fe26b8fb640 10 monclient: get_monmap_and_config 2024-03-29T11:20:56.896 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.918+0000 7fe26b8fb640 10 monclient: build_initial_monmap 2024-03-29T11:20:56.896 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.918+0000 7fe26b8fb640 10 monclient: monmap: 2024-03-29T11:20:56.896 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:20:56.896 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:20:56.896 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:20:56.920251+0000 2024-03-29T11:20:56.896 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:20:56.920251+0000 2024-03-29T11:20:56.896 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:56.896 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:56.896 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:20:56.896 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:20:56.896 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:20:56.896 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:56.900 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.922+0000 7fe26b8fb640 1 Processor -- start 2024-03-29T11:20:56.900 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.923+0000 7fe26b8fb640 1 -- start start 2024-03-29T11:20:56.900 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.923+0000 7fe26b8fb640 10 monclient: init 2024-03-29T11:20:56.901 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.924+0000 7fe26b8fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:56.901 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.924+0000 7fe26b8fb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:20:56.906 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.927+0000 7fe26b8fb640 10 monclient(hunting): picked mon.noname-c con 0x7fe2641457f0 addr v1:172.21.15.94:6789/0 2024-03-29T11:20:56.906 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.927+0000 7fe26b8fb640 10 monclient(hunting): picked mon.noname-a con 0x7fe26413f650 addr v1:172.21.15.17:6789/0 2024-03-29T11:20:56.906 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.927+0000 7fe26b8fb640 10 monclient(hunting): picked mon.noname-b con 0x7fe2640a9d70 addr v1:172.21.15.17:6790/0 2024-03-29T11:20:56.906 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.927+0000 7fe26b8fb640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2640aaef0 con 0x7fe26413f650 2024-03-29T11:20:56.906 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.927+0000 7fe26b8fb640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe264145bd0 con 0x7fe2641457f0 2024-03-29T11:20:56.906 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.927+0000 7fe26b8fb640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe264145eb0 con 0x7fe2640a9d70 2024-03-29T11:20:56.906 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.927+0000 7fe26b8fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:56.906 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.927+0000 7fe26b8fb640 10 monclient(hunting): authenticate will time out at 963.133766s 2024-03-29T11:20:56.909 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.931+0000 7fe269697640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fe2641457f0 0x7fe26413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:60933/0 (socket says 172.21.15.17:60933) 2024-03-29T11:20:56.909 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.931+0000 7fe269697640 1 -- 172.21.15.17:0/4082297199 learned_addr learned my addr 172.21.15.17:0/4082297199 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:20:56.910 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.932+0000 7fe25bfff640 1 -- 172.21.15.17:0/4082297199 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4246420147 0 0) 0x7fe2640aaef0 con 0x7fe26413f650 2024-03-29T11:20:56.911 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.932+0000 7fe25bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:56.911 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.932+0000 7fe25bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:56.911 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.932+0000 7fe25bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:56.911 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.932+0000 7fe25bfff640 10 monclient(hunting): my global_id is 4296 2024-03-29T11:20:56.911 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.932+0000 7fe25bfff640 1 -- 172.21.15.17:0/4082297199 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe250003520 con 0x7fe26413f650 2024-03-29T11:20:56.911 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.933+0000 7fe25bfff640 1 -- 172.21.15.17:0/4082297199 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3570281312 0 0) 0x7fe250003520 con 0x7fe26413f650 2024-03-29T11:20:56.911 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.933+0000 7fe25bfff640 1 -- 172.21.15.17:0/4082297199 >> v1:172.21.15.17:6790/0 conn(0x7fe2640a9d70 legacy=0x7fe26414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:56.911 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.933+0000 7fe25bfff640 1 -- 172.21.15.17:0/4082297199 >> v1:172.21.15.94:6789/0 conn(0x7fe2641457f0 legacy=0x7fe26413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:20:56.911 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.933+0000 7fe25bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:20:56.911 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.933+0000 7fe25bfff640 1 monclient: found mon.noname-a 2024-03-29T11:20:56.911 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.933+0000 7fe25bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:56.911 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.933+0000 7fe25bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.17:6789/0 2024-03-29T11:20:56.911 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.933+0000 7fe25bfff640 1 -- 172.21.15.17:0/4082297199 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe264146340 con 0x7fe26413f650 2024-03-29T11:20:56.911 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.933+0000 7fe25bfff640 10 monclient: _finish_auth 0 2024-03-29T11:20:56.912 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.933+0000 7fe25bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:20:56.912 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.933+0000 7fe25bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:56.912 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.933+0000 7fe25bfff640 1 -- 172.21.15.17:0/4082297199 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fe260001690 con 0x7fe26413f650 2024-03-29T11:20:56.912 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.933+0000 7fe25bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:56.912 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.933+0000 7fe25bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:56.912 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.933+0000 7fe25bfff640 10 monclient: dump: 2024-03-29T11:20:56.912 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:20:56.912 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:20:56.912 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:56.912 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:56.912 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:56.912 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:56.912 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:20:56.912 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:20:56.913 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:20:56.913 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:56.913 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.933+0000 7fe25bfff640 1 -- 172.21.15.17:0/4082297199 <== mon.0 v1:172.21.15.17:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe260001860 con 0x7fe26413f650 2024-03-29T11:20:56.913 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.933+0000 7fe25bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:56.913 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.933+0000 7fe25bfff640 1 -- 172.21.15.17:0/4082297199 <== mon.0 v1:172.21.15.17:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fe260001cb0 con 0x7fe26413f650 2024-03-29T11:20:56.913 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.933+0000 7fe25bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:56.913 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.933+0000 7fe25bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:56.913 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.933+0000 7fe25bfff640 10 monclient: dump: 2024-03-29T11:20:56.913 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:20:56.913 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:20:56.913 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:56.913 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:56.913 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:56.914 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:56.914 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:20:56.914 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:20:56.914 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:20:56.914 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:56.914 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.933+0000 7fe26b8fb640 5 monclient: authenticate success, global_id 4296 2024-03-29T11:20:56.914 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.933+0000 7fe26b8fb640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:56.914 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.933+0000 7fe26b8fb640 4 set_mon_vals no callback set 2024-03-29T11:20:56.914 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.934+0000 7fe26b8fb640 10 monclient: shutdown 2024-03-29T11:20:56.914 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.934+0000 7fe26b8fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:56.915 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.934+0000 7fe26b8fb640 1 -- 172.21.15.17:0/4082297199 >> v1:172.21.15.17:6789/0 conn(0x7fe26413f650 legacy=0x7fe2640b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:56.915 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.938+0000 7fe26b8fb640 1 -- 172.21.15.17:0/4082297199 shutdown_connections 2024-03-29T11:20:56.915 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.938+0000 7fe26b8fb640 1 -- 172.21.15.17:0/4082297199 >> 172.21.15.17:0/4082297199 conn(0x7fe264066a00 msgr2=0x7fe264066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:56.918 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.941+0000 7fe26b8fb640 1 -- 172.21.15.17:0/4082297199 shutdown_connections 2024-03-29T11:20:56.919 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.942+0000 7fe26b8fb640 1 -- 172.21.15.17:0/4082297199 wait complete. 2024-03-29T11:20:56.923 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.943+0000 7fe26b8fb640 10 monclient: build_initial_monmap 2024-03-29T11:20:56.923 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.943+0000 7fe26b8fb640 10 monclient: monmap: 2024-03-29T11:20:56.923 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:20:56.923 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:56.923 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:20:56.923 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:20:56.923 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:56.923 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:56.923 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:20:56.923 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:20:56.923 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:20:56.923 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:56.923 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.943+0000 7fe26b8fb640 1 librados: starting msgr at 2024-03-29T11:20:56.924 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.943+0000 7fe26b8fb640 1 librados: starting objecter 2024-03-29T11:20:56.924 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.944+0000 7fe26b8fb640 1 Processor -- start 2024-03-29T11:20:56.924 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.944+0000 7fe26b8fb640 1 -- start start 2024-03-29T11:20:56.924 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.944+0000 7fe26b8fb640 1 librados: setting wanted keys 2024-03-29T11:20:56.924 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.944+0000 7fe26b8fb640 1 librados: calling monclient init 2024-03-29T11:20:56.924 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.944+0000 7fe26b8fb640 10 monclient: init 2024-03-29T11:20:56.924 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.945+0000 7fe26b8fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:56.924 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.945+0000 7fe26b8fb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:20:56.925 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.948+0000 7fe26b8fb640 10 monclient(hunting): picked mon.noname-c con 0x7fe2640a9d70 addr v1:172.21.15.17:6790/0 2024-03-29T11:20:56.926 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.948+0000 7fe269697640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7fe2640a9d70 0x7fe2640ab600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:34523/0 (socket says 172.21.15.17:34523) 2024-03-29T11:20:56.926 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.948+0000 7fe269697640 1 -- 172.21.15.17:0/2842957807 learned_addr learned my addr 172.21.15.17:0/2842957807 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:20:56.926 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.948+0000 7fe259ffb640 10 client.?.objecter ms_handle_connect 0x7fe2640a9d70 2024-03-29T11:20:56.926 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.948+0000 7fe259ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:56.926 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.948+0000 7fe26b8fb640 10 monclient(hunting): picked mon.noname-b con 0x7fe26413f650 addr v1:172.21.15.94:6789/0 2024-03-29T11:20:56.928 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.948+0000 7fe26b8fb640 10 monclient(hunting): picked mon.noname-a con 0x7fe2641457f0 addr v1:172.21.15.17:6789/0 2024-03-29T11:20:56.928 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.950+0000 7fe26b8fb640 1 -- 172.21.15.17:0/2842957807 --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2641db340 con 0x7fe2641457f0 2024-03-29T11:20:56.928 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.950+0000 7fe26b8fb640 1 -- 172.21.15.17:0/2842957807 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2641387a0 con 0x7fe26413f650 2024-03-29T11:20:56.928 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.950+0000 7fe26b8fb640 1 -- 172.21.15.17:0/2842957807 --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2640ad7a0 con 0x7fe2640a9d70 2024-03-29T11:20:56.928 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.950+0000 7fe26b8fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:56.928 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.950+0000 7fe26b8fb640 10 monclient(hunting): authenticate will time out at 963.156121s 2024-03-29T11:20:56.928 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.951+0000 7fe259ffb640 1 -- 172.21.15.17:0/2842957807 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2660813352 0 0) 0x7fe2640ad7a0 con 0x7fe2640a9d70 2024-03-29T11:20:56.930 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.952+0000 7fe259ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:56.930 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.952+0000 7fe259ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:56.930 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.952+0000 7fe259ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:56.930 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.952+0000 7fe259ffb640 10 monclient(hunting): my global_id is 4307 2024-03-29T11:20:56.930 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.952+0000 7fe259ffb640 1 -- 172.21.15.17:0/2842957807 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe248004eb0 con 0x7fe2640a9d70 2024-03-29T11:20:56.934 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.955+0000 7fe259ffb640 1 -- 172.21.15.17:0/2842957807 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1251772586 0 0) 0x7fe248004eb0 con 0x7fe2640a9d70 2024-03-29T11:20:56.934 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.955+0000 7fe259ffb640 1 -- 172.21.15.17:0/2842957807 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2640ad7a0 con 0x7fe2640a9d70 2024-03-29T11:20:56.934 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.955+0000 7fe259ffb640 1 -- 172.21.15.17:0/2842957807 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fe25c001b50 con 0x7fe2640a9d70 2024-03-29T11:20:56.935 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.955+0000 7fe259ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:56.935 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.955+0000 7fe259ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:56.935 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.955+0000 7fe259ffb640 10 monclient(hunting): dump: 2024-03-29T11:20:56.935 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:20:56.935 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:20:56.935 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:56.935 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:56.935 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:56.935 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:56.935 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:20:56.935 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:20:56.935 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:20:56.935 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:56.935 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.955+0000 7fe259ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:56.935 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.955+0000 7fe259ffb640 1 -- 172.21.15.17:0/2842957807 <== mon.2 v1:172.21.15.17:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3948696604 0 0) 0x7fe2640ad7a0 con 0x7fe2640a9d70 2024-03-29T11:20:56.936 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.955+0000 7fe259ffb640 1 -- 172.21.15.17:0/2842957807 >> v1:172.21.15.94:6789/0 conn(0x7fe26413f650 legacy=0x7fe264137980 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:20:56.936 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.955+0000 7fe259ffb640 1 -- 172.21.15.17:0/2842957807 >> v1:172.21.15.17:6789/0 conn(0x7fe2641457f0 legacy=0x7fe264138090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:56.936 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.955+0000 7fe259ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:20:56.936 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.955+0000 7fe259ffb640 1 monclient: found mon.c 2024-03-29T11:20:56.936 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.955+0000 7fe259ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:56.936 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.955+0000 7fe259ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:20:56.936 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.955+0000 7fe259ffb640 1 -- 172.21.15.17:0/2842957807 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2641db7d0 con 0x7fe2640a9d70 2024-03-29T11:20:56.936 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.956+0000 7fe259ffb640 10 monclient: _finish_auth 0 2024-03-29T11:20:56.936 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.956+0000 7fe259ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:20:56.936 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.956+0000 7fe259ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:56.936 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.956+0000 7fe26b8fb640 5 monclient: authenticate success, global_id 4307 2024-03-29T11:20:56.936 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.956+0000 7fe26b8fb640 10 monclient: _renew_subs 2024-03-29T11:20:56.936 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.956+0000 7fe26b8fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:20:56.936 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.956+0000 7fe26b8fb640 1 -- 172.21.15.17:0/2842957807 --> v1:172.21.15.17:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2640abd10 con 0x7fe2640a9d70 2024-03-29T11:20:56.936 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.956+0000 7fe26b8fb640 10 client.4307.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:56.937 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.956+0000 7fe26b8fb640 10 monclient: _renew_subs 2024-03-29T11:20:56.937 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.956+0000 7fe26b8fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:20:56.937 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.956+0000 7fe26b8fb640 1 -- 172.21.15.17:0/2842957807 --> v1:172.21.15.17:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe264142d10 con 0x7fe2640a9d70 2024-03-29T11:20:56.937 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.956+0000 7fe26b8fb640 1 librados: init done 2024-03-29T11:20:56.938 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.958+0000 7fe26b8fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:56.938 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.958+0000 7fe26b8fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:56.938 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.958+0000 7fe26b8fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:20:56.938 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.958+0000 7fe26b8fb640 1 -- 172.21.15.17:0/2842957807 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe264075f10 con 0x7fe2640a9d70 2024-03-29T11:20:56.938 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.958+0000 7fe259ffb640 1 -- 172.21.15.17:0/2842957807 <== mon.2 v1:172.21.15.17:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe25c001cb0 con 0x7fe2640a9d70 2024-03-29T11:20:56.938 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.958+0000 7fe259ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:56.938 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.958+0000 7fe25b7fe640 4 set_mon_vals no callback set 2024-03-29T11:20:56.938 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.959+0000 7fe259ffb640 1 -- 172.21.15.17:0/2842957807 <== mon.2 v1:172.21.15.17:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fe25c004e00 con 0x7fe2640a9d70 2024-03-29T11:20:56.938 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.959+0000 7fe259ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:56.938 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.959+0000 7fe259ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:56.939 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.959+0000 7fe259ffb640 10 monclient: dump: 2024-03-29T11:20:56.939 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:20:56.939 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:20:56.939 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:56.939 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:20:56.939 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:56.939 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:20:56.939 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:20:56.939 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:20:56.939 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:20:56.939 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:20:56.939 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.959+0000 7fe259ffb640 1 -- 172.21.15.17:0/2842957807 <== mon.2 v1:172.21.15.17:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7fe25c01ae90 con 0x7fe2640a9d70 2024-03-29T11:20:56.941 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.959+0000 7fe259ffb640 1 -- 172.21.15.17:0/2842957807 <== mon.2 v1:172.21.15.17:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7fe25c084f30 con 0x7fe2640a9d70 2024-03-29T11:20:56.941 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.959+0000 7fe259ffb640 10 client.4307.objecter ms_dispatch 0x7fe264000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:56.941 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.959+0000 7fe259ffb640 3 client.4307.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:56.941 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.959+0000 7fe259ffb640 3 client.4307.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:56.941 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.960+0000 7fe259ffb640 20 client.4307.objecter dump_active .. 0 homeless 2024-03-29T11:20:56.941 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.960+0000 7fe268e96640 10 monclient: get_auth_request con 0x7fe24806dcd0 auth_method 0 2024-03-29T11:20:56.941 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.960+0000 7fe259ffb640 10 client.4307.objecter ms_handle_connect 0x7fe24806dcd0 2024-03-29T11:20:56.944 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.966+0000 7fe259ffb640 1 -- 172.21.15.17:0/2842957807 <== mon.2 v1:172.21.15.17:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe25c0506c0 con 0x7fe2640a9d70 2024-03-29T11:20:56.944 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.966+0000 7fe259ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:56.944 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:56.966+0000 7fe259ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:57.272 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:57.294+0000 7fe26b8fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:57.272 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:57.294+0000 7fe26b8fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:57.272 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:57.295+0000 7fe26b8fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:20:57.272 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:57.295+0000 7fe26b8fb640 1 -- 172.21.15.17:0/2842957807 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe2640b3680 con 0x7fe2640a9d70 2024-03-29T11:20:57.273 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:57.295+0000 7fe259ffb640 1 -- 172.21.15.17:0/2842957807 <== mon.2 v1:172.21.15.17:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7fe25c0527c0 con 0x7fe2640a9d70 2024-03-29T11:20:57.273 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:57.295+0000 7fe259ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:57.273 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:57.295+0000 7fe259ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:57.273 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:20:57.273 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:57.276 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:57.299+0000 7fe26b8fb640 10 librados: watch_flush enter 2024-03-29T11:20:57.276 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:57.299+0000 7fe26b8fb640 10 librados: watch_flush exit 2024-03-29T11:20:57.276 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:57.299+0000 7fe26b8fb640 20 client.4307.objecter shutdown clearing up homeless session... 2024-03-29T11:20:57.276 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:57.299+0000 7fe26b8fb640 10 client.4307.objecter successfully canceled tick 2024-03-29T11:20:57.277 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:57.299+0000 7fe26b8fb640 1 -- 172.21.15.17:0/2842957807 >> v1:172.21.15.17:6816/3236185052 conn(0x7fe24806dcd0 legacy=0x7fe248070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:57.277 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:57.299+0000 7fe26b8fb640 10 monclient: shutdown 2024-03-29T11:20:57.277 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:57.299+0000 7fe26b8fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:57.278 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:57.299+0000 7fe26b8fb640 1 -- 172.21.15.17:0/2842957807 >> v1:172.21.15.17:6790/0 conn(0x7fe2640a9d70 legacy=0x7fe2640ab600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:57.278 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:57.301+0000 7fe26b8fb640 1 -- 172.21.15.17:0/2842957807 shutdown_connections 2024-03-29T11:20:57.278 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:57.301+0000 7fe26b8fb640 1 -- 172.21.15.17:0/2842957807 >> 172.21.15.17:0/2842957807 conn(0x7fe264066a00 msgr2=0x7fe2640b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:57.278 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:57.301+0000 7fe26b8fb640 1 -- 172.21.15.17:0/2842957807 shutdown_connections 2024-03-29T11:20:57.279 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:57.301+0000 7fe26b8fb640 1 -- 172.21.15.17:0/2842957807 wait complete. 2024-03-29T11:20:57.279 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:20:57.301+0000 7fe26b8fb640 1 librados: shutdown 2024-03-29T11:20:57.295 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:04.103 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:04.263 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.286+0000 7f7dca90e640 10 monclient: get_monmap_and_config 2024-03-29T11:21:04.263 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.286+0000 7f7dca90e640 10 monclient: build_initial_monmap 2024-03-29T11:21:04.263 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.286+0000 7f7dca90e640 10 monclient: monmap: 2024-03-29T11:21:04.263 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:21:04.263 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:21:04.263 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:21:04.287554+0000 2024-03-29T11:21:04.263 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:21:04.287554+0000 2024-03-29T11:21:04.264 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:04.264 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:04.264 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:21:04.264 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:21:04.264 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:21:04.264 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:04.265 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.287+0000 7f7dca90e640 1 Processor -- start 2024-03-29T11:21:04.265 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.287+0000 7f7dca90e640 1 -- start start 2024-03-29T11:21:04.265 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.287+0000 7f7dca90e640 10 monclient: init 2024-03-29T11:21:04.265 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.288+0000 7f7dca90e640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:04.265 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.288+0000 7f7dca90e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:04.268 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.291+0000 7f7dca90e640 10 monclient(hunting): picked mon.noname-b con 0x7f7dc40b1f60 addr v1:172.21.15.17:6790/0 2024-03-29T11:21:04.268 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.291+0000 7f7dca90e640 10 monclient(hunting): picked mon.noname-c con 0x7f7dc4071eb0 addr v1:172.21.15.94:6789/0 2024-03-29T11:21:04.268 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.291+0000 7f7dca90e640 10 monclient(hunting): picked mon.noname-a con 0x7f7dc4143d10 addr v1:172.21.15.17:6789/0 2024-03-29T11:21:04.268 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.291+0000 7f7dca90e640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dc406ef80 con 0x7f7dc4143d10 2024-03-29T11:21:04.269 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.291+0000 7f7dca90e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dc406f260 con 0x7f7dc4071eb0 2024-03-29T11:21:04.269 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.291+0000 7f7dca90e640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dc40b40c0 con 0x7f7dc40b1f60 2024-03-29T11:21:04.269 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.291+0000 7f7dca90e640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:04.269 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.291+0000 7f7dca90e640 10 monclient(hunting): authenticate will time out at 970.496282s 2024-03-29T11:21:04.271 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.294+0000 7f7dc3fff640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f7dc40b1f60 0x7f7dc4143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:55055/0 (socket says 172.21.15.17:55055) 2024-03-29T11:21:04.271 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.294+0000 7f7dc3fff640 1 -- 172.21.15.17:0/3377292261 learned_addr learned my addr 172.21.15.17:0/3377292261 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:21:04.272 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.295+0000 7f7dc2ffd640 1 -- 172.21.15.17:0/3377292261 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 339008769 0 0) 0x7f7dc40b40c0 con 0x7f7dc40b1f60 2024-03-29T11:21:04.272 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.295+0000 7f7dc2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:04.272 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.295+0000 7f7dc2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:04.272 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.295+0000 7f7dc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:04.272 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.295+0000 7f7dc2ffd640 10 monclient(hunting): my global_id is 4310 2024-03-29T11:21:04.272 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.295+0000 7f7dc2ffd640 1 -- 172.21.15.17:0/3377292261 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dac003520 con 0x7f7dc40b1f60 2024-03-29T11:21:04.272 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.295+0000 7f7dc2ffd640 1 -- 172.21.15.17:0/3377292261 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 149998708 0 0) 0x7f7dc406ef80 con 0x7f7dc4143d10 2024-03-29T11:21:04.272 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.295+0000 7f7dc2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:04.272 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.295+0000 7f7dc2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:04.272 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.295+0000 7f7dc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:04.272 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.295+0000 7f7dc2ffd640 10 monclient(hunting): my global_id is 4299 2024-03-29T11:21:04.272 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.295+0000 7f7dc2ffd640 1 -- 172.21.15.17:0/3377292261 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dc40b40c0 con 0x7f7dc4143d10 2024-03-29T11:21:04.273 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.295+0000 7f7dc2ffd640 1 -- 172.21.15.17:0/3377292261 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2164736190 0 0) 0x7f7dc406f260 con 0x7f7dc4071eb0 2024-03-29T11:21:04.273 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.295+0000 7f7dc2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:04.273 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.295+0000 7f7dc2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:04.273 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.295+0000 7f7dc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:04.273 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.295+0000 7f7dc2ffd640 10 monclient(hunting): my global_id is 4309 2024-03-29T11:21:04.273 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.295+0000 7f7dc2ffd640 1 -- 172.21.15.17:0/3377292261 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dac0039d0 con 0x7f7dc4071eb0 2024-03-29T11:21:04.275 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.296+0000 7f7dc2ffd640 1 -- 172.21.15.17:0/3377292261 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1762741806 0 0) 0x7f7dc40b40c0 con 0x7f7dc4143d10 2024-03-29T11:21:04.275 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.296+0000 7f7dc2ffd640 1 -- 172.21.15.17:0/3377292261 >> v1:172.21.15.17:6790/0 conn(0x7f7dc40b1f60 legacy=0x7f7dc4143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:04.275 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.296+0000 7f7dc2ffd640 1 -- 172.21.15.17:0/3377292261 >> v1:172.21.15.94:6789/0 conn(0x7f7dc4071eb0 legacy=0x7f7dc4146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:04.275 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.296+0000 7f7dc2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:21:04.275 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.296+0000 7f7dc2ffd640 1 monclient: found mon.noname-a 2024-03-29T11:21:04.275 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.296+0000 7f7dc2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:04.275 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.296+0000 7f7dc2ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.17:6789/0 2024-03-29T11:21:04.275 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.296+0000 7f7dc2ffd640 1 -- 172.21.15.17:0/3377292261 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc4155d70 con 0x7f7dc4143d10 2024-03-29T11:21:04.276 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.296+0000 7f7dc2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:21:04.276 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.296+0000 7f7dc2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:21:04.276 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.296+0000 7f7dc2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:04.276 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.296+0000 7f7dc2ffd640 1 -- 172.21.15.17:0/3377292261 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f7dbc001970 con 0x7f7dc4143d10 2024-03-29T11:21:04.276 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.296+0000 7f7dc2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:04.276 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.296+0000 7f7dc2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:04.276 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.296+0000 7f7dc2ffd640 10 monclient: dump: 2024-03-29T11:21:04.276 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:21:04.276 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:21:04.276 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:04.276 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:04.276 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:04.276 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:04.276 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:21:04.277 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:21:04.277 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:21:04.277 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:04.277 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.296+0000 7f7dca90e640 5 monclient: authenticate success, global_id 4299 2024-03-29T11:21:04.277 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.296+0000 7f7dca90e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:04.277 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.296+0000 7f7dc2ffd640 1 -- 172.21.15.17:0/3377292261 <== mon.0 v1:172.21.15.17:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7dbc001b50 con 0x7f7dc4143d10 2024-03-29T11:21:04.277 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.296+0000 7f7dc2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:04.277 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.296+0000 7f7dca90e640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:04.277 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.296+0000 7f7dca90e640 4 set_mon_vals no callback set 2024-03-29T11:21:04.277 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.297+0000 7f7dca90e640 10 monclient: shutdown 2024-03-29T11:21:04.277 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.297+0000 7f7dca90e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:04.277 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.297+0000 7f7dca90e640 1 -- 172.21.15.17:0/3377292261 >> v1:172.21.15.17:6789/0 conn(0x7f7dc4143d10 legacy=0x7f7dc4146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:04.277 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.297+0000 7f7dc2ffd640 1 -- 172.21.15.17:0/3377292261 <== mon.0 v1:172.21.15.17:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f7dbc001970 con 0x7f7dc4143d10 2024-03-29T11:21:04.278 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.297+0000 7f7dc2ffd640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:21:04.278 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.297+0000 7f7dca90e640 1 -- 172.21.15.17:0/3377292261 shutdown_connections 2024-03-29T11:21:04.278 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.297+0000 7f7dca90e640 1 -- 172.21.15.17:0/3377292261 >> 172.21.15.17:0/3377292261 conn(0x7f7dc4066a00 msgr2=0x7f7dc4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:04.278 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.298+0000 7f7dca90e640 1 -- 172.21.15.17:0/3377292261 shutdown_connections 2024-03-29T11:21:04.278 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.298+0000 7f7dca90e640 1 -- 172.21.15.17:0/3377292261 wait complete. 2024-03-29T11:21:04.279 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.299+0000 7f7dca90e640 10 monclient: build_initial_monmap 2024-03-29T11:21:04.280 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.299+0000 7f7dca90e640 10 monclient: monmap: 2024-03-29T11:21:04.280 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:21:04.280 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:04.280 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:21:04.280 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:21:04.280 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:04.280 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:04.280 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:21:04.280 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:21:04.280 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:21:04.280 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:04.280 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.299+0000 7f7dca90e640 1 librados: starting msgr at 2024-03-29T11:21:04.280 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.300+0000 7f7dca90e640 1 librados: starting objecter 2024-03-29T11:21:04.280 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.300+0000 7f7dca90e640 1 Processor -- start 2024-03-29T11:21:04.281 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.301+0000 7f7dca90e640 1 -- start start 2024-03-29T11:21:04.281 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.301+0000 7f7dca90e640 1 librados: setting wanted keys 2024-03-29T11:21:04.281 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.301+0000 7f7dca90e640 1 librados: calling monclient init 2024-03-29T11:21:04.281 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.301+0000 7f7dca90e640 10 monclient: init 2024-03-29T11:21:04.281 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.302+0000 7f7dca90e640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:04.281 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.302+0000 7f7dca90e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:21:04.281 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.302+0000 7f7dca90e640 10 monclient(hunting): picked mon.noname-a con 0x7f7dc40b1f60 addr v1:172.21.15.17:6789/0 2024-03-29T11:21:04.281 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.302+0000 7f7dca90e640 10 monclient(hunting): picked mon.noname-b con 0x7f7dc4143d10 addr v1:172.21.15.94:6789/0 2024-03-29T11:21:04.281 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.302+0000 7f7dca90e640 10 monclient(hunting): picked mon.noname-c con 0x7f7dc413a650 addr v1:172.21.15.17:6790/0 2024-03-29T11:21:04.281 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.302+0000 7f7dca90e640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dc413c4f0 con 0x7f7dc40b1f60 2024-03-29T11:21:04.281 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.302+0000 7f7dca90e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dc413c760 con 0x7f7dc4143d10 2024-03-29T11:21:04.281 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.302+0000 7f7dca90e640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dc413ca40 con 0x7f7dc413a650 2024-03-29T11:21:04.281 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.302+0000 7f7dca90e640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:04.281 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.302+0000 7f7dca90e640 10 monclient(hunting): authenticate will time out at 970.507723s 2024-03-29T11:21:04.287 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.309+0000 7f7dc3fff640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f7dc40b1f60 0x7f7dc41def00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:46865/0 (socket says 172.21.15.17:46865) 2024-03-29T11:21:04.287 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.309+0000 7f7dc3fff640 1 -- 172.21.15.17:0/1997185781 learned_addr learned my addr 172.21.15.17:0/1997185781 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:21:04.288 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.310+0000 7f7dc0ff9640 10 client.?.objecter ms_handle_connect 0x7f7dc40b1f60 2024-03-29T11:21:04.288 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.310+0000 7f7dc0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:04.288 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.310+0000 7f7dc0ff9640 1 -- 172.21.15.17:0/1997185781 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 176537836 0 0) 0x7f7dc413c4f0 con 0x7f7dc40b1f60 2024-03-29T11:21:04.288 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.310+0000 7f7dc0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:21:04.288 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.310+0000 7f7dc0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:04.288 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.310+0000 7f7dc0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:04.288 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.310+0000 7f7dc0ff9640 10 monclient(hunting): my global_id is 4302 2024-03-29T11:21:04.288 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.310+0000 7f7dc0ff9640 1 -- 172.21.15.17:0/1997185781 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7db0004eb0 con 0x7f7dc40b1f60 2024-03-29T11:21:04.288 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.310+0000 7f7dc0ff9640 1 -- 172.21.15.17:0/1997185781 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1943644308 0 0) 0x7f7db0004eb0 con 0x7f7dc40b1f60 2024-03-29T11:21:04.288 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.310+0000 7f7dc0ff9640 1 -- 172.21.15.17:0/1997185781 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7db0005300 con 0x7f7dc40b1f60 2024-03-29T11:21:04.288 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.310+0000 7f7dc0ff9640 1 -- 172.21.15.17:0/1997185781 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f7db8001950 con 0x7f7dc40b1f60 2024-03-29T11:21:04.289 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.310+0000 7f7dc0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:04.289 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.310+0000 7f7dc0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:04.289 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.310+0000 7f7dc0ff9640 10 monclient(hunting): dump: 2024-03-29T11:21:04.289 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:21:04.289 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:21:04.289 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:04.289 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:04.289 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:04.289 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:04.289 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:21:04.289 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:21:04.289 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:21:04.289 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:04.290 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.310+0000 7f7dc0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:04.290 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.310+0000 7f7dc0ff9640 10 client.?.objecter ms_handle_connect 0x7f7dc413a650 2024-03-29T11:21:04.290 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.310+0000 7f7dc0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:04.290 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.311+0000 7f7dc0ff9640 1 -- 172.21.15.17:0/1997185781 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 301376449 0 0) 0x7f7dc413ca40 con 0x7f7dc413a650 2024-03-29T11:21:04.290 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.311+0000 7f7dc0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:21:04.290 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.311+0000 7f7dc0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:04.290 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.311+0000 7f7dc0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:04.290 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.311+0000 7f7dc0ff9640 10 monclient(hunting): my global_id is 4313 2024-03-29T11:21:04.290 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.311+0000 7f7dc0ff9640 1 -- 172.21.15.17:0/1997185781 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7db0004eb0 con 0x7f7dc413a650 2024-03-29T11:21:04.290 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.311+0000 7f7dc0ff9640 1 -- 172.21.15.17:0/1997185781 <== mon.0 v1:172.21.15.17:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1853317029 0 0) 0x7f7db0005300 con 0x7f7dc40b1f60 2024-03-29T11:21:04.291 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.311+0000 7f7dc0ff9640 1 -- 172.21.15.17:0/1997185781 >> v1:172.21.15.17:6790/0 conn(0x7f7dc413a650 legacy=0x7f7dc41e6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:04.291 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.311+0000 7f7dc0ff9640 1 -- 172.21.15.17:0/1997185781 >> v1:172.21.15.94:6789/0 conn(0x7f7dc4143d10 legacy=0x7f7dc41df610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:04.291 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.311+0000 7f7dc0ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:21:04.291 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.311+0000 7f7dc0ff9640 1 monclient: found mon.a 2024-03-29T11:21:04.291 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.311+0000 7f7dc0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:04.291 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.311+0000 7f7dc0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:21:04.291 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.311+0000 7f7dc0ff9640 1 -- 172.21.15.17:0/1997185781 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc413b760 con 0x7f7dc40b1f60 2024-03-29T11:21:04.291 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.311+0000 7f7dc0ff9640 10 monclient: _finish_auth 0 2024-03-29T11:21:04.291 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.311+0000 7f7dc0ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:21:04.291 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.311+0000 7f7dc0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:04.292 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.311+0000 7f7dca90e640 5 monclient: authenticate success, global_id 4302 2024-03-29T11:21:04.292 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.311+0000 7f7dca90e640 10 monclient: _renew_subs 2024-03-29T11:21:04.292 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.311+0000 7f7dca90e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:21:04.292 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.311+0000 7f7dca90e640 1 -- 172.21.15.17:0/1997185781 --> v1:172.21.15.17:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dc413b110 con 0x7f7dc40b1f60 2024-03-29T11:21:04.292 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.311+0000 7f7dca90e640 10 client.4302.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:04.293 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.311+0000 7f7dca90e640 10 monclient: _renew_subs 2024-03-29T11:21:04.293 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.311+0000 7f7dca90e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:21:04.293 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.311+0000 7f7dca90e640 1 -- 172.21.15.17:0/1997185781 --> v1:172.21.15.17:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dc41e7a20 con 0x7f7dc40b1f60 2024-03-29T11:21:04.293 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.311+0000 7f7dca90e640 1 librados: init done 2024-03-29T11:21:04.293 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.313+0000 7f7dca90e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:04.293 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.313+0000 7f7dca90e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:04.293 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.313+0000 7f7dca90e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:21:04.293 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.313+0000 7f7dca90e640 1 -- 172.21.15.17:0/1997185781 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dc413b5a0 con 0x7f7dc40b1f60 2024-03-29T11:21:04.293 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.315+0000 7f7dc0ff9640 1 -- 172.21.15.17:0/1997185781 <== mon.0 v1:172.21.15.17:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7db80047e0 con 0x7f7dc40b1f60 2024-03-29T11:21:04.297 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.315+0000 7f7dc0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:04.297 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.315+0000 7f7dc27fc640 4 set_mon_vals no callback set 2024-03-29T11:21:04.297 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.315+0000 7f7dc0ff9640 1 -- 172.21.15.17:0/1997185781 <== mon.0 v1:172.21.15.17:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f7db8004bb0 con 0x7f7dc40b1f60 2024-03-29T11:21:04.297 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.316+0000 7f7dc0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:04.297 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.316+0000 7f7dc0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:04.297 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.316+0000 7f7dc0ff9640 10 monclient: dump: 2024-03-29T11:21:04.298 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:21:04.298 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:21:04.298 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:04.298 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:04.298 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:04.298 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:04.298 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:21:04.298 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:21:04.298 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:21:04.298 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:04.298 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.316+0000 7f7dc0ff9640 1 -- 172.21.15.17:0/1997185781 <== mon.0 v1:172.21.15.17:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f7db8004dd0 con 0x7f7dc40b1f60 2024-03-29T11:21:04.299 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.316+0000 7f7dc0ff9640 1 -- 172.21.15.17:0/1997185781 <== mon.0 v1:172.21.15.17:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f7db8084b50 con 0x7f7dc40b1f60 2024-03-29T11:21:04.299 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.316+0000 7f7dc0ff9640 10 client.4302.objecter ms_dispatch 0x7f7dc4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:04.299 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.316+0000 7f7dc0ff9640 3 client.4302.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:04.299 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.316+0000 7f7dc0ff9640 3 client.4302.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:04.299 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.317+0000 7f7dc0ff9640 20 client.4302.objecter dump_active .. 0 homeless 2024-03-29T11:21:04.299 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.317+0000 7f7dc37fe640 10 monclient: get_auth_request con 0x7f7db006df60 auth_method 0 2024-03-29T11:21:04.299 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.317+0000 7f7dc0ff9640 10 client.4302.objecter ms_handle_connect 0x7f7db006df60 2024-03-29T11:21:04.300 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.321+0000 7f7dc0ff9640 1 -- 172.21.15.17:0/1997185781 <== mon.0 v1:172.21.15.17:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7db80504b0 con 0x7f7dc40b1f60 2024-03-29T11:21:04.300 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.321+0000 7f7dc0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:04.300 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.321+0000 7f7dc0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:04.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.645+0000 7f7dca90e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:04.622 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.645+0000 7f7dca90e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:04.622 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.645+0000 7f7dca90e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:21:04.622 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.645+0000 7f7dca90e640 1 -- 172.21.15.17:0/1997185781 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7dc4075f10 con 0x7f7dc40b1f60 2024-03-29T11:21:04.624 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.647+0000 7f7dc0ff9640 1 -- 172.21.15.17:0/1997185781 <== mon.0 v1:172.21.15.17:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f7db80523e0 con 0x7f7dc40b1f60 2024-03-29T11:21:04.624 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.647+0000 7f7dc0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:04.624 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.647+0000 7f7dc0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:04.626 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:21:04.626 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:04.629 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.652+0000 7f7dca90e640 10 librados: watch_flush enter 2024-03-29T11:21:04.629 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.652+0000 7f7dca90e640 10 librados: watch_flush exit 2024-03-29T11:21:04.629 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.652+0000 7f7dca90e640 20 client.4302.objecter shutdown clearing up homeless session... 2024-03-29T11:21:04.629 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.652+0000 7f7dca90e640 10 client.4302.objecter successfully canceled tick 2024-03-29T11:21:04.630 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.653+0000 7f7dca90e640 1 -- 172.21.15.17:0/1997185781 >> v1:172.21.15.17:6816/3236185052 conn(0x7f7db006df60 legacy=0x7f7db0070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:04.631 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.653+0000 7f7dca90e640 10 monclient: shutdown 2024-03-29T11:21:04.631 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.653+0000 7f7dca90e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:04.631 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.653+0000 7f7dca90e640 1 -- 172.21.15.17:0/1997185781 >> v1:172.21.15.17:6789/0 conn(0x7f7dc40b1f60 legacy=0x7f7dc41def00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:04.631 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.653+0000 7f7dca90e640 1 -- 172.21.15.17:0/1997185781 shutdown_connections 2024-03-29T11:21:04.631 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.653+0000 7f7dca90e640 1 -- 172.21.15.17:0/1997185781 >> 172.21.15.17:0/1997185781 conn(0x7f7dc4066a00 msgr2=0x7f7dc4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:04.631 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.654+0000 7f7dca90e640 1 -- 172.21.15.17:0/1997185781 shutdown_connections 2024-03-29T11:21:04.631 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.654+0000 7f7dca90e640 1 -- 172.21.15.17:0/1997185781 wait complete. 2024-03-29T11:21:04.631 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:04.654+0000 7f7dca90e640 1 librados: shutdown 2024-03-29T11:21:04.647 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:11.454 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:11.610 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.632+0000 7fda0b91e640 10 monclient: get_monmap_and_config 2024-03-29T11:21:11.611 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.632+0000 7fda0b91e640 10 monclient: build_initial_monmap 2024-03-29T11:21:11.611 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.632+0000 7fda0b91e640 10 monclient: monmap: 2024-03-29T11:21:11.611 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:21:11.611 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:21:11.611 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:21:11.633932+0000 2024-03-29T11:21:11.611 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:21:11.633932+0000 2024-03-29T11:21:11.611 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:11.611 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:11.611 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:21:11.611 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:21:11.611 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:21:11.611 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:11.611 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.634+0000 7fda0b91e640 1 Processor -- start 2024-03-29T11:21:11.612 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.634+0000 7fda0b91e640 1 -- start start 2024-03-29T11:21:11.612 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.634+0000 7fda0b91e640 10 monclient: init 2024-03-29T11:21:11.612 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.635+0000 7fda0b91e640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:11.612 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.635+0000 7fda0b91e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:21:11.612 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.635+0000 7fda0b91e640 10 monclient(hunting): picked mon.noname-a con 0x7fda040b4100 addr v1:172.21.15.17:6789/0 2024-03-29T11:21:11.613 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.635+0000 7fda0b91e640 10 monclient(hunting): picked mon.noname-b con 0x7fda0413ee90 addr v1:172.21.15.17:6790/0 2024-03-29T11:21:11.613 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.635+0000 7fda0b91e640 10 monclient(hunting): picked mon.noname-c con 0x7fda040af900 addr v1:172.21.15.94:6789/0 2024-03-29T11:21:11.613 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.635+0000 7fda0b91e640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda040afd10 con 0x7fda040b4100 2024-03-29T11:21:11.613 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.635+0000 7fda0b91e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda040af6f0 con 0x7fda040af900 2024-03-29T11:21:11.613 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.635+0000 7fda0b91e640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda040aaf60 con 0x7fda0413ee90 2024-03-29T11:21:11.613 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.635+0000 7fda0b91e640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:11.613 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.635+0000 7fda0b91e640 10 monclient(hunting): authenticate will time out at 977.840585s 2024-03-29T11:21:11.613 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.636+0000 7fda096ba640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fda040b4100 0x7fda040a9850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:44692/0 (socket says 172.21.15.17:44692) 2024-03-29T11:21:11.613 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.636+0000 7fda096ba640 1 -- 172.21.15.17:0/1292576267 learned_addr learned my addr 172.21.15.17:0/1292576267 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:21:11.613 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.636+0000 7fd9fbfff640 1 -- 172.21.15.17:0/1292576267 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4038298387 0 0) 0x7fda040af6f0 con 0x7fda040af900 2024-03-29T11:21:11.613 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.636+0000 7fd9fbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:11.613 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.636+0000 7fd9fbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:11.613 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.636+0000 7fd9fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:11.613 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.636+0000 7fd9fbfff640 10 monclient(hunting): my global_id is 4312 2024-03-29T11:21:11.613 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.636+0000 7fd9fbfff640 1 -- 172.21.15.17:0/1292576267 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9f0003520 con 0x7fda040af900 2024-03-29T11:21:11.616 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.636+0000 7fd9fbfff640 1 -- 172.21.15.17:0/1292576267 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 701618809 0 0) 0x7fda040afd10 con 0x7fda040b4100 2024-03-29T11:21:11.616 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.636+0000 7fd9fbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:11.616 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.636+0000 7fd9fbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:11.617 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.636+0000 7fd9fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:11.617 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.636+0000 7fd9fbfff640 10 monclient(hunting): my global_id is 4305 2024-03-29T11:21:11.617 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.636+0000 7fd9fbfff640 1 -- 172.21.15.17:0/1292576267 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda040af6f0 con 0x7fda040b4100 2024-03-29T11:21:11.617 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.637+0000 7fd9fbfff640 1 -- 172.21.15.17:0/1292576267 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 199171002 0 0) 0x7fd9f0003520 con 0x7fda040af900 2024-03-29T11:21:11.617 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.637+0000 7fd9fbfff640 1 -- 172.21.15.17:0/1292576267 >> v1:172.21.15.17:6790/0 conn(0x7fda0413ee90 legacy=0x7fda040adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:11.617 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.637+0000 7fd9fbfff640 1 -- 172.21.15.17:0/1292576267 >> v1:172.21.15.17:6789/0 conn(0x7fda040b4100 legacy=0x7fda040a9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:11.617 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.637+0000 7fd9fbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:21:11.617 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.637+0000 7fd9fbfff640 1 monclient: found mon.noname-c 2024-03-29T11:21:11.617 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.637+0000 7fd9fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:11.617 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.637+0000 7fd9fbfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6789/0 2024-03-29T11:21:11.618 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.637+0000 7fd9fbfff640 1 -- 172.21.15.17:0/1292576267 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda0413e970 con 0x7fda040af900 2024-03-29T11:21:11.618 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.637+0000 7fd9fbfff640 10 monclient: _finish_auth 0 2024-03-29T11:21:11.618 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.637+0000 7fd9fbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:21:11.618 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.637+0000 7fd9fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:11.618 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.637+0000 7fda0b91e640 5 monclient: authenticate success, global_id 4312 2024-03-29T11:21:11.618 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.637+0000 7fda0b91e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:11.618 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.637+0000 7fd9fbfff640 1 -- 172.21.15.17:0/1292576267 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fd9fc001a10 con 0x7fda040af900 2024-03-29T11:21:11.618 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.637+0000 7fd9fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:11.618 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.637+0000 7fd9fbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:11.618 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.637+0000 7fd9fbfff640 10 monclient: dump: 2024-03-29T11:21:11.618 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:21:11.619 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:21:11.619 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:11.619 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:11.619 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:11.619 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:11.619 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:21:11.619 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:21:11.619 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:21:11.619 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:11.619 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.637+0000 7fda0b91e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:11.619 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.638+0000 7fd9fbfff640 1 -- 172.21.15.17:0/1292576267 <== mon.1 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9fc001b70 con 0x7fda040af900 2024-03-29T11:21:11.619 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.638+0000 7fd9fbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:11.619 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.638+0000 7fda0b91e640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:11.620 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.638+0000 7fda0b91e640 4 set_mon_vals no callback set 2024-03-29T11:21:11.620 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.638+0000 7fd9fbfff640 1 -- 172.21.15.17:0/1292576267 <== mon.1 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fd9fc001fc0 con 0x7fda040af900 2024-03-29T11:21:11.620 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.638+0000 7fd9fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:11.620 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.638+0000 7fd9fbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:11.620 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.638+0000 7fd9fbfff640 10 monclient: dump: 2024-03-29T11:21:11.620 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:21:11.620 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:21:11.620 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:11.620 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:11.620 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:11.620 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:11.620 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:21:11.620 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:21:11.620 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:21:11.621 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:11.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.639+0000 7fda0b91e640 10 monclient: shutdown 2024-03-29T11:21:11.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.639+0000 7fda0b91e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:11.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.639+0000 7fda0b91e640 1 -- 172.21.15.17:0/1292576267 >> v1:172.21.15.94:6789/0 conn(0x7fda040af900 legacy=0x7fda0413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:11.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.639+0000 7fda0b91e640 1 -- 172.21.15.17:0/1292576267 shutdown_connections 2024-03-29T11:21:11.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.639+0000 7fda0b91e640 1 -- 172.21.15.17:0/1292576267 >> 172.21.15.17:0/1292576267 conn(0x7fda040b35e0 msgr2=0x7fda040b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:11.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.640+0000 7fda0b91e640 1 -- 172.21.15.17:0/1292576267 shutdown_connections 2024-03-29T11:21:11.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.640+0000 7fda0b91e640 1 -- 172.21.15.17:0/1292576267 wait complete. 2024-03-29T11:21:11.628 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.641+0000 7fda0b91e640 10 monclient: build_initial_monmap 2024-03-29T11:21:11.628 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.641+0000 7fda0b91e640 10 monclient: monmap: 2024-03-29T11:21:11.628 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:21:11.628 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:11.628 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:21:11.628 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:21:11.628 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:11.628 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:11.629 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:21:11.629 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:21:11.629 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:21:11.629 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:11.629 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.641+0000 7fda0b91e640 1 librados: starting msgr at 2024-03-29T11:21:11.629 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.641+0000 7fda0b91e640 1 librados: starting objecter 2024-03-29T11:21:11.629 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.641+0000 7fda0b91e640 1 Processor -- start 2024-03-29T11:21:11.629 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.641+0000 7fda0b91e640 1 -- start start 2024-03-29T11:21:11.629 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.641+0000 7fda0b91e640 1 librados: setting wanted keys 2024-03-29T11:21:11.629 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.641+0000 7fda0b91e640 1 librados: calling monclient init 2024-03-29T11:21:11.629 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.641+0000 7fda0b91e640 10 monclient: init 2024-03-29T11:21:11.629 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.642+0000 7fda0b91e640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:11.629 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.642+0000 7fda0b91e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:21:11.629 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.642+0000 7fda0b91e640 10 monclient(hunting): picked mon.noname-a con 0x7fda040af900 addr v1:172.21.15.17:6789/0 2024-03-29T11:21:11.630 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.642+0000 7fda096ba640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fda040af900 0x7fda04130ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:44898/0 (socket says 172.21.15.17:44898) 2024-03-29T11:21:11.630 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.642+0000 7fda096ba640 1 -- 172.21.15.17:0/3052373845 learned_addr learned my addr 172.21.15.17:0/3052373845 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:21:11.630 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.643+0000 7fda0b91e640 10 monclient(hunting): picked mon.noname-b con 0x7fda040b4100 addr v1:172.21.15.94:6789/0 2024-03-29T11:21:11.630 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.643+0000 7fda0b91e640 10 monclient(hunting): picked mon.noname-c con 0x7fda0413ee90 addr v1:172.21.15.17:6790/0 2024-03-29T11:21:11.630 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.643+0000 7fda0b91e640 1 -- 172.21.15.17:0/3052373845 --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda0413d190 con 0x7fda040af900 2024-03-29T11:21:11.630 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.643+0000 7fda0b91e640 1 -- 172.21.15.17:0/3052373845 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda0413cd60 con 0x7fda040b4100 2024-03-29T11:21:11.630 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.643+0000 7fda0b91e640 1 -- 172.21.15.17:0/3052373845 --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda04130580 con 0x7fda0413ee90 2024-03-29T11:21:11.630 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.643+0000 7fda0b91e640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:11.630 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.643+0000 7fda0b91e640 10 monclient(hunting): authenticate will time out at 977.848310s 2024-03-29T11:21:11.630 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.643+0000 7fd9f9ffb640 10 client.?.objecter ms_handle_connect 0x7fda040af900 2024-03-29T11:21:11.630 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.643+0000 7fd9f9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:11.630 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.643+0000 7fd9f9ffb640 10 client.?.objecter ms_handle_connect 0x7fda040b4100 2024-03-29T11:21:11.630 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.643+0000 7fd9f9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:11.630 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.643+0000 7fd9f9ffb640 10 client.?.objecter ms_handle_connect 0x7fda0413ee90 2024-03-29T11:21:11.631 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.643+0000 7fd9f9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:11.631 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.644+0000 7fd9f9ffb640 1 -- 172.21.15.17:0/3052373845 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1485104601 0 0) 0x7fda0413cd60 con 0x7fda040b4100 2024-03-29T11:21:11.631 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.644+0000 7fd9f9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:11.631 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.644+0000 7fd9f9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:11.631 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.644+0000 7fd9f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:11.631 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.644+0000 7fd9f9ffb640 10 monclient(hunting): my global_id is 4315 2024-03-29T11:21:11.631 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.644+0000 7fd9f9ffb640 1 -- 172.21.15.17:0/3052373845 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9e8004eb0 con 0x7fda040b4100 2024-03-29T11:21:11.631 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.644+0000 7fd9f9ffb640 1 -- 172.21.15.17:0/3052373845 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 741611701 0 0) 0x7fda0413d190 con 0x7fda040af900 2024-03-29T11:21:11.631 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.644+0000 7fd9f9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:11.631 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.644+0000 7fd9f9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:11.632 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.644+0000 7fd9f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:11.632 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.644+0000 7fd9f9ffb640 10 monclient(hunting): my global_id is 4308 2024-03-29T11:21:11.632 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.644+0000 7fd9f9ffb640 1 -- 172.21.15.17:0/3052373845 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda0413cd60 con 0x7fda040af900 2024-03-29T11:21:11.632 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.644+0000 7fd9f9ffb640 1 -- 172.21.15.17:0/3052373845 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4288634705 0 0) 0x7fda04130580 con 0x7fda0413ee90 2024-03-29T11:21:11.632 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.644+0000 7fd9f9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:11.632 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.644+0000 7fd9f9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:11.632 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.644+0000 7fd9f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:11.632 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.644+0000 7fd9f9ffb640 10 monclient(hunting): my global_id is 4319 2024-03-29T11:21:11.632 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.644+0000 7fd9f9ffb640 1 -- 172.21.15.17:0/3052373845 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda0413d190 con 0x7fda0413ee90 2024-03-29T11:21:11.632 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.645+0000 7fd9f9ffb640 1 -- 172.21.15.17:0/3052373845 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 234586691 0 0) 0x7fda0413cd60 con 0x7fda040af900 2024-03-29T11:21:11.632 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.645+0000 7fd9f9ffb640 1 -- 172.21.15.17:0/3052373845 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda04130580 con 0x7fda040af900 2024-03-29T11:21:11.632 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.645+0000 7fd9f9ffb640 1 -- 172.21.15.17:0/3052373845 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fd9f40019f0 con 0x7fda040af900 2024-03-29T11:21:11.632 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.645+0000 7fd9f9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:11.632 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.645+0000 7fd9f9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:11.632 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.645+0000 7fd9f9ffb640 10 monclient(hunting): dump: 2024-03-29T11:21:11.633 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:21:11.633 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:21:11.633 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:11.633 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:11.633 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:11.633 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:11.633 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:21:11.633 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:21:11.633 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:21:11.633 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:11.633 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.645+0000 7fd9f9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:11.633 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.645+0000 7fd9f9ffb640 1 -- 172.21.15.17:0/3052373845 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1307254281 0 0) 0x7fd9e8004eb0 con 0x7fda040b4100 2024-03-29T11:21:11.633 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.645+0000 7fd9f9ffb640 1 -- 172.21.15.17:0/3052373845 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda0413cd60 con 0x7fda040b4100 2024-03-29T11:21:11.633 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.645+0000 7fd9f9ffb640 1 -- 172.21.15.17:0/3052373845 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fda00001850 con 0x7fda040b4100 2024-03-29T11:21:11.634 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.645+0000 7fd9f9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:11.634 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.645+0000 7fd9f9ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:11.634 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.645+0000 7fd9f9ffb640 10 monclient(hunting): dump: 2024-03-29T11:21:11.634 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:21:11.634 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:21:11.634 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:11.634 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:11.634 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:11.634 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:11.634 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:21:11.634 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:21:11.634 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:21:11.634 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:11.635 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.645+0000 7fd9f9ffb640 1 -- 172.21.15.17:0/3052373845 <== mon.0 v1:172.21.15.17:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 626608874 0 0) 0x7fda04130580 con 0x7fda040af900 2024-03-29T11:21:11.635 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.645+0000 7fd9f9ffb640 1 -- 172.21.15.17:0/3052373845 >> v1:172.21.15.17:6790/0 conn(0x7fda0413ee90 legacy=0x7fda04121880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:11.635 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.645+0000 7fd9f9ffb640 1 -- 172.21.15.17:0/3052373845 >> v1:172.21.15.94:6789/0 conn(0x7fda040b4100 legacy=0x7fda04121170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:11.635 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.645+0000 7fd9f9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:21:11.635 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.645+0000 7fd9f9ffb640 1 monclient: found mon.a 2024-03-29T11:21:11.635 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.645+0000 7fd9f9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:11.635 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.645+0000 7fd9f9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:21:11.635 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.645+0000 7fd9f9ffb640 1 -- 172.21.15.17:0/3052373845 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda04130860 con 0x7fda040af900 2024-03-29T11:21:11.635 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.646+0000 7fd9f9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:21:11.635 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.646+0000 7fd9f9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:21:11.635 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.646+0000 7fd9f9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:11.635 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.646+0000 7fda0b91e640 5 monclient: authenticate success, global_id 4308 2024-03-29T11:21:11.635 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.646+0000 7fda0b91e640 10 monclient: _renew_subs 2024-03-29T11:21:11.635 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.646+0000 7fda0b91e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:21:11.635 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.646+0000 7fda0b91e640 1 -- 172.21.15.17:0/3052373845 --> v1:172.21.15.17:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda0413d3e0 con 0x7fda040af900 2024-03-29T11:21:11.636 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.646+0000 7fda0b91e640 10 client.4308.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:11.636 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.646+0000 7fda0b91e640 10 monclient: _renew_subs 2024-03-29T11:21:11.636 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.646+0000 7fda0b91e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:21:11.636 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.646+0000 7fda0b91e640 1 -- 172.21.15.17:0/3052373845 --> v1:172.21.15.17:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fda0413de70 con 0x7fda040af900 2024-03-29T11:21:11.636 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.646+0000 7fda0b91e640 1 librados: init done 2024-03-29T11:21:11.636 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.647+0000 7fda0b91e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:11.636 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.647+0000 7fda0b91e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:11.636 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.647+0000 7fda0b91e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:21:11.636 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.647+0000 7fda0b91e640 1 -- 172.21.15.17:0/3052373845 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda04075f10 con 0x7fda040af900 2024-03-29T11:21:11.636 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.648+0000 7fd9f9ffb640 1 -- 172.21.15.17:0/3052373845 <== mon.0 v1:172.21.15.17:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9f40047e0 con 0x7fda040af900 2024-03-29T11:21:11.636 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.648+0000 7fd9f9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:11.636 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.648+0000 7fd9fb7fe640 4 set_mon_vals no callback set 2024-03-29T11:21:11.636 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.649+0000 7fd9f9ffb640 1 -- 172.21.15.17:0/3052373845 <== mon.0 v1:172.21.15.17:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fd9f4004b70 con 0x7fda040af900 2024-03-29T11:21:11.637 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.649+0000 7fd9f9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:11.637 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.649+0000 7fd9f9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:11.637 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.649+0000 7fd9f9ffb640 10 monclient: dump: 2024-03-29T11:21:11.637 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:21:11.637 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:21:11.637 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:11.637 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:11.637 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:11.637 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:11.637 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:21:11.637 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:21:11.638 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:21:11.638 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:11.638 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.649+0000 7fd9f9ffb640 1 -- 172.21.15.17:0/3052373845 <== mon.0 v1:172.21.15.17:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7fd9f4004d90 con 0x7fda040af900 2024-03-29T11:21:11.638 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.650+0000 7fd9f9ffb640 1 -- 172.21.15.17:0/3052373845 <== mon.0 v1:172.21.15.17:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7fd9f40849f0 con 0x7fda040af900 2024-03-29T11:21:11.638 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.650+0000 7fd9f9ffb640 10 client.4308.objecter ms_dispatch 0x7fda04000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:11.638 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.650+0000 7fd9f9ffb640 3 client.4308.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:11.638 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.650+0000 7fd9f9ffb640 3 client.4308.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:11.638 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.650+0000 7fda08eb9640 10 monclient: get_auth_request con 0x7fd9e806dfd0 auth_method 0 2024-03-29T11:21:11.638 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.650+0000 7fd9f9ffb640 20 client.4308.objecter dump_active .. 0 homeless 2024-03-29T11:21:11.638 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.651+0000 7fd9f9ffb640 10 client.4308.objecter ms_handle_connect 0x7fd9e806dfd0 2024-03-29T11:21:11.640 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.663+0000 7fd9f9ffb640 1 -- 172.21.15.17:0/3052373845 <== mon.0 v1:172.21.15.17:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd9f4050390 con 0x7fda040af900 2024-03-29T11:21:11.640 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.663+0000 7fd9f9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:11.640 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:11.663+0000 7fd9f9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:12.069 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:12.093+0000 7fda0b91e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:12.070 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:12.093+0000 7fda0b91e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:12.070 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:12.093+0000 7fda0b91e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:21:12.070 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:12.093+0000 7fda0b91e640 1 -- 172.21.15.17:0/3052373845 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda040b2f30 con 0x7fda040af900 2024-03-29T11:21:12.070 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:12.094+0000 7fd9f9ffb640 1 -- 172.21.15.17:0/3052373845 <== mon.0 v1:172.21.15.17:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7fd9f40522c0 con 0x7fda040af900 2024-03-29T11:21:12.071 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:12.094+0000 7fd9f9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:12.071 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:12.094+0000 7fd9f9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:12.072 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:21:12.072 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:12.075 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:12.097+0000 7fda0b91e640 10 librados: watch_flush enter 2024-03-29T11:21:12.075 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:12.097+0000 7fda0b91e640 10 librados: watch_flush exit 2024-03-29T11:21:12.075 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:12.097+0000 7fda0b91e640 20 client.4308.objecter shutdown clearing up homeless session... 2024-03-29T11:21:12.075 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:12.097+0000 7fda0b91e640 10 client.4308.objecter successfully canceled tick 2024-03-29T11:21:12.075 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:12.098+0000 7fda0b91e640 1 -- 172.21.15.17:0/3052373845 >> v1:172.21.15.17:6816/3236185052 conn(0x7fd9e806dfd0 legacy=0x7fd9e8070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:12.075 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:12.098+0000 7fda0b91e640 10 monclient: shutdown 2024-03-29T11:21:12.075 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:12.098+0000 7fda0b91e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:12.075 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:12.098+0000 7fda0b91e640 1 -- 172.21.15.17:0/3052373845 >> v1:172.21.15.17:6789/0 conn(0x7fda040af900 legacy=0x7fda04130ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:12.075 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:12.098+0000 7fda0b91e640 1 -- 172.21.15.17:0/3052373845 shutdown_connections 2024-03-29T11:21:12.075 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:12.098+0000 7fda0b91e640 1 -- 172.21.15.17:0/3052373845 >> 172.21.15.17:0/3052373845 conn(0x7fda040b35e0 msgr2=0x7fda04125fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:12.076 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:12.098+0000 7fda0b91e640 1 -- 172.21.15.17:0/3052373845 shutdown_connections 2024-03-29T11:21:12.076 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:12.098+0000 7fda0b91e640 1 -- 172.21.15.17:0/3052373845 wait complete. 2024-03-29T11:21:12.076 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:12.098+0000 7fda0b91e640 1 librados: shutdown 2024-03-29T11:21:12.086 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:18.893 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:19.039 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.062+0000 7f2c42f15640 10 monclient: get_monmap_and_config 2024-03-29T11:21:19.039 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.062+0000 7f2c42f15640 10 monclient: build_initial_monmap 2024-03-29T11:21:19.039 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.062+0000 7f2c42f15640 10 monclient: monmap: 2024-03-29T11:21:19.039 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:21:19.039 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:21:19.039 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:21:19.063735+0000 2024-03-29T11:21:19.040 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:21:19.063735+0000 2024-03-29T11:21:19.040 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:19.040 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:19.040 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:21:19.040 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:21:19.040 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:21:19.040 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:19.041 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.063+0000 7f2c42f15640 1 Processor -- start 2024-03-29T11:21:19.041 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.064+0000 7f2c42f15640 1 -- start start 2024-03-29T11:21:19.041 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.064+0000 7f2c42f15640 10 monclient: init 2024-03-29T11:21:19.041 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.064+0000 7f2c42f15640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:19.041 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.064+0000 7f2c42f15640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:21:19.041 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.064+0000 7f2c42f15640 10 monclient(hunting): picked mon.noname-c con 0x7f2c3c06cac0 addr v1:172.21.15.94:6789/0 2024-03-29T11:21:19.041 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.064+0000 7f2c42f15640 10 monclient(hunting): picked mon.noname-b con 0x7f2c3c06cea0 addr v1:172.21.15.17:6790/0 2024-03-29T11:21:19.042 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.064+0000 7f2c42f15640 10 monclient(hunting): picked mon.noname-a con 0x7f2c3c145e80 addr v1:172.21.15.17:6789/0 2024-03-29T11:21:19.042 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.064+0000 7f2c42f15640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c3c0ac850 con 0x7f2c3c145e80 2024-03-29T11:21:19.042 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.065+0000 7f2c42f15640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c3c070470 con 0x7f2c3c06cac0 2024-03-29T11:21:19.042 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.065+0000 7f2c42f15640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c3c072600 con 0x7f2c3c06cea0 2024-03-29T11:21:19.042 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.065+0000 7f2c42f15640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:19.042 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.065+0000 7f2c42f15640 10 monclient(hunting): authenticate will time out at 985.269903s 2024-03-29T11:21:19.042 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.065+0000 7f2c3bfff640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f2c3c06cea0 0x7f2c3c145770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:45746/0 (socket says 172.21.15.17:45746) 2024-03-29T11:21:19.042 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.065+0000 7f2c40cb1640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f2c3c06cac0 0x7f2c3c071ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:39956/0 (socket says 172.21.15.17:39956) 2024-03-29T11:21:19.042 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.065+0000 7f2c3bfff640 1 -- 172.21.15.17:0/1764030726 learned_addr learned my addr 172.21.15.17:0/1764030726 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:21:19.045 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.065+0000 7f2c3b7fe640 1 -- 172.21.15.17:0/1764030726 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2062134784 0 0) 0x7f2c3c072600 con 0x7f2c3c06cea0 2024-03-29T11:21:19.046 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.065+0000 7f2c3b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:19.046 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.065+0000 7f2c3b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:19.046 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.066+0000 7f2c3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:19.046 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.066+0000 7f2c3b7fe640 10 monclient(hunting): my global_id is 4322 2024-03-29T11:21:19.046 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.066+0000 7f2c3b7fe640 1 -- 172.21.15.17:0/1764030726 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c2c003520 con 0x7f2c3c06cea0 2024-03-29T11:21:19.046 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.066+0000 7f2c3b7fe640 1 -- 172.21.15.17:0/1764030726 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3988846627 0 0) 0x7f2c3c070470 con 0x7f2c3c06cac0 2024-03-29T11:21:19.046 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.066+0000 7f2c3b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:19.046 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.066+0000 7f2c3b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:19.046 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.066+0000 7f2c3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:19.046 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.066+0000 7f2c3b7fe640 10 monclient(hunting): my global_id is 4318 2024-03-29T11:21:19.047 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.066+0000 7f2c3b7fe640 1 -- 172.21.15.17:0/1764030726 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c3c072600 con 0x7f2c3c06cac0 2024-03-29T11:21:19.047 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.066+0000 7f2c3b7fe640 1 -- 172.21.15.17:0/1764030726 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2168692558 0 0) 0x7f2c3c0ac850 con 0x7f2c3c145e80 2024-03-29T11:21:19.047 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.066+0000 7f2c3b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:19.047 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.066+0000 7f2c3b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:19.047 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.066+0000 7f2c3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:19.047 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.066+0000 7f2c3b7fe640 10 monclient(hunting): my global_id is 4311 2024-03-29T11:21:19.047 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.066+0000 7f2c3b7fe640 1 -- 172.21.15.17:0/1764030726 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c3c070470 con 0x7f2c3c145e80 2024-03-29T11:21:19.047 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.066+0000 7f2c3b7fe640 1 -- 172.21.15.17:0/1764030726 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3343248889 0 0) 0x7f2c3c072600 con 0x7f2c3c06cac0 2024-03-29T11:21:19.047 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.066+0000 7f2c3b7fe640 1 -- 172.21.15.17:0/1764030726 >> v1:172.21.15.17:6790/0 conn(0x7f2c3c06cea0 legacy=0x7f2c3c145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:19.047 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.066+0000 7f2c3b7fe640 1 -- 172.21.15.17:0/1764030726 >> v1:172.21.15.17:6789/0 conn(0x7f2c3c145e80 legacy=0x7f2c3c14db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:19.047 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.066+0000 7f2c3b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:21:19.047 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.066+0000 7f2c3b7fe640 1 monclient: found mon.noname-c 2024-03-29T11:21:19.047 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.066+0000 7f2c3b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:19.048 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.066+0000 7f2c3b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6789/0 2024-03-29T11:21:19.048 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.067+0000 7f2c3b7fe640 1 -- 172.21.15.17:0/1764030726 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c3c1462d0 con 0x7f2c3c06cac0 2024-03-29T11:21:19.048 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.067+0000 7f2c3b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:21:19.048 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.067+0000 7f2c3b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:21:19.048 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.067+0000 7f2c3b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:19.048 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.067+0000 7f2c42f15640 5 monclient: authenticate success, global_id 4318 2024-03-29T11:21:19.048 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.067+0000 7f2c42f15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:19.048 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.067+0000 7f2c3b7fe640 1 -- 172.21.15.17:0/1764030726 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f2c300015f0 con 0x7f2c3c06cac0 2024-03-29T11:21:19.048 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.067+0000 7f2c3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:19.048 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.067+0000 7f2c3b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:19.048 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.067+0000 7f2c3b7fe640 10 monclient: dump: 2024-03-29T11:21:19.049 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:21:19.049 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:21:19.049 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:19.049 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:19.049 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:19.049 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:19.049 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:21:19.049 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:21:19.049 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:21:19.049 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:19.049 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.067+0000 7f2c42f15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:19.049 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.067+0000 7f2c3b7fe640 1 -- 172.21.15.17:0/1764030726 <== mon.1 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2c30001810 con 0x7f2c3c06cac0 2024-03-29T11:21:19.049 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.067+0000 7f2c3b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:19.049 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.067+0000 7f2c3b7fe640 1 -- 172.21.15.17:0/1764030726 <== mon.1 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f2c30001ba0 con 0x7f2c3c06cac0 2024-03-29T11:21:19.050 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.067+0000 7f2c3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:19.050 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.067+0000 7f2c3b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:19.050 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.067+0000 7f2c3b7fe640 10 monclient: dump: 2024-03-29T11:21:19.050 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:21:19.050 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:21:19.050 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:19.050 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:19.050 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:19.050 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:19.050 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:21:19.050 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:21:19.051 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:21:19.051 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:19.051 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.067+0000 7f2c42f15640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:19.051 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.067+0000 7f2c42f15640 4 set_mon_vals no callback set 2024-03-29T11:21:19.051 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.068+0000 7f2c42f15640 10 monclient: shutdown 2024-03-29T11:21:19.051 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.068+0000 7f2c42f15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:19.051 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.068+0000 7f2c42f15640 1 -- 172.21.15.17:0/1764030726 >> v1:172.21.15.94:6789/0 conn(0x7f2c3c06cac0 legacy=0x7f2c3c071ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:19.051 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.068+0000 7f2c42f15640 1 -- 172.21.15.17:0/1764030726 shutdown_connections 2024-03-29T11:21:19.051 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.068+0000 7f2c42f15640 1 -- 172.21.15.17:0/1764030726 >> 172.21.15.17:0/1764030726 conn(0x7f2c3c0675b0 msgr2=0x7f2c3c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:19.051 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.068+0000 7f2c42f15640 1 -- 172.21.15.17:0/1764030726 shutdown_connections 2024-03-29T11:21:19.051 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.068+0000 7f2c42f15640 1 -- 172.21.15.17:0/1764030726 wait complete. 2024-03-29T11:21:19.052 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.069+0000 7f2c42f15640 10 monclient: build_initial_monmap 2024-03-29T11:21:19.052 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.069+0000 7f2c42f15640 10 monclient: monmap: 2024-03-29T11:21:19.052 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:21:19.052 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:19.052 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:21:19.052 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:21:19.052 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:19.052 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:19.052 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:21:19.053 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:21:19.053 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:21:19.053 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:19.060 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.070+0000 7f2c42f15640 1 librados: starting msgr at 2024-03-29T11:21:19.061 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.070+0000 7f2c42f15640 1 librados: starting objecter 2024-03-29T11:21:19.061 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.070+0000 7f2c42f15640 1 Processor -- start 2024-03-29T11:21:19.061 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.070+0000 7f2c42f15640 1 -- start start 2024-03-29T11:21:19.061 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.070+0000 7f2c42f15640 1 librados: setting wanted keys 2024-03-29T11:21:19.061 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.070+0000 7f2c42f15640 1 librados: calling monclient init 2024-03-29T11:21:19.061 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.070+0000 7f2c42f15640 10 monclient: init 2024-03-29T11:21:19.061 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.072+0000 7f2c42f15640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:19.061 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.072+0000 7f2c42f15640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:21:19.061 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.072+0000 7f2c42f15640 10 monclient(hunting): picked mon.noname-b con 0x7f2c3c06cac0 addr v1:172.21.15.94:6789/0 2024-03-29T11:21:19.061 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.072+0000 7f2c42f15640 10 monclient(hunting): picked mon.noname-a con 0x7f2c3c06cea0 addr v1:172.21.15.17:6789/0 2024-03-29T11:21:19.061 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.072+0000 7f2c3bfff640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f2c3c06cea0 0x7f2c3c1d7ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:58404/0 (socket says 172.21.15.17:58404) 2024-03-29T11:21:19.061 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.073+0000 7f2c3bfff640 1 -- 172.21.15.17:0/1079526867 learned_addr learned my addr 172.21.15.17:0/1079526867 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:21:19.061 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.073+0000 7f2c42f15640 10 monclient(hunting): picked mon.noname-c con 0x7f2c3c145e80 addr v1:172.21.15.17:6790/0 2024-03-29T11:21:19.062 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.073+0000 7f2c42f15640 1 -- 172.21.15.17:0/1079526867 --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c3c0aeb70 con 0x7f2c3c06cea0 2024-03-29T11:21:19.062 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.073+0000 7f2c397fa640 10 client.?.objecter ms_handle_connect 0x7f2c3c06cea0 2024-03-29T11:21:19.062 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.073+0000 7f2c397fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:19.062 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.073+0000 7f2c42f15640 1 -- 172.21.15.17:0/1079526867 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c3c1d50d0 con 0x7f2c3c06cac0 2024-03-29T11:21:19.062 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.073+0000 7f2c42f15640 1 -- 172.21.15.17:0/1079526867 --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c3c0afe70 con 0x7f2c3c145e80 2024-03-29T11:21:19.062 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.073+0000 7f2c42f15640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:19.062 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.073+0000 7f2c42f15640 10 monclient(hunting): authenticate will time out at 985.278705s 2024-03-29T11:21:19.062 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.073+0000 7f2c40cb1640 1 -- 172.21.15.17:0/1079526867 >> v1:172.21.15.94:6789/0 conn(0x7f2c3c06cac0 legacy=0x7f2c3c0ae460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-29T11:21:19.062 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.073+0000 7f2c40cb1640 1 -- 172.21.15.17:0/1079526867 >> v1:172.21.15.94:6789/0 conn(0x7f2c3c06cac0 legacy=0x7f2c3c0ae460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:21:19.062 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.073+0000 7f2c397fa640 10 client.?.objecter ms_handle_connect 0x7f2c3c06cac0 2024-03-29T11:21:19.062 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.073+0000 7f2c397fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:19.062 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.073+0000 7f2c40cb1640 1 --1- 172.21.15.17:0/1079526867 >> v1:172.21.15.94:6789/0 conn(0x7f2c3c06cac0 0x7f2c3c0ae460 :-1 s=OPENED pgs=309400 cs=1 l=1).handle_message read tag failed 2024-03-29T11:21:19.062 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.073+0000 7f2c40cb1640 1 --1- 172.21.15.17:0/1079526867 >> v1:172.21.15.94:6789/0 conn(0x7f2c3c06cac0 0x7f2c3c0ae460 :-1 s=OPENED pgs=309400 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:21:19.062 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.073+0000 7f2c397fa640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.94:6789/0 2024-03-29T11:21:19.063 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.074+0000 7f2c397fa640 10 client.?.objecter ms_handle_connect 0x7f2c3c145e80 2024-03-29T11:21:19.063 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.074+0000 7f2c397fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:19.063 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.074+0000 7f2c397fa640 1 -- 172.21.15.17:0/1079526867 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2070863947 0 0) 0x7f2c3c0aeb70 con 0x7f2c3c06cea0 2024-03-29T11:21:19.063 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.074+0000 7f2c397fa640 10 monclient(hunting): _negotiate 2024-03-29T11:21:19.063 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.074+0000 7f2c397fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:19.063 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.074+0000 7f2c397fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:19.063 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.074+0000 7f2c397fa640 10 monclient(hunting): my global_id is 4314 2024-03-29T11:21:19.063 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.074+0000 7f2c397fa640 1 -- 172.21.15.17:0/1079526867 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c20004eb0 con 0x7f2c3c06cea0 2024-03-29T11:21:19.063 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.074+0000 7f2c397fa640 1 -- 172.21.15.17:0/1079526867 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2603432322 0 0) 0x7f2c3c0afe70 con 0x7f2c3c145e80 2024-03-29T11:21:19.063 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.074+0000 7f2c397fa640 10 monclient(hunting): _negotiate 2024-03-29T11:21:19.063 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.074+0000 7f2c397fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:19.063 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.074+0000 7f2c397fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:19.063 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.074+0000 7f2c397fa640 10 monclient(hunting): my global_id is 4325 2024-03-29T11:21:19.063 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.074+0000 7f2c397fa640 1 -- 172.21.15.17:0/1079526867 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c200051d0 con 0x7f2c3c145e80 2024-03-29T11:21:19.063 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.074+0000 7f2c397fa640 1 -- 172.21.15.17:0/1079526867 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1035045024 0 0) 0x7f2c20004eb0 con 0x7f2c3c06cea0 2024-03-29T11:21:19.064 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.074+0000 7f2c397fa640 1 -- 172.21.15.17:0/1079526867 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c20005640 con 0x7f2c3c06cea0 2024-03-29T11:21:19.064 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.074+0000 7f2c397fa640 1 -- 172.21.15.17:0/1079526867 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f2c28001990 con 0x7f2c3c06cea0 2024-03-29T11:21:19.064 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.074+0000 7f2c397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:19.064 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.074+0000 7f2c397fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:19.064 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.074+0000 7f2c397fa640 10 monclient(hunting): dump: 2024-03-29T11:21:19.064 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:21:19.064 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:21:19.064 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:19.064 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:19.064 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:19.064 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:19.064 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:21:19.064 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:21:19.064 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:21:19.065 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:19.065 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.074+0000 7f2c397fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:19.065 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.074+0000 7f2c397fa640 1 -- 172.21.15.17:0/1079526867 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2604950654 0 0) 0x7f2c200051d0 con 0x7f2c3c145e80 2024-03-29T11:21:19.065 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.074+0000 7f2c397fa640 1 -- 172.21.15.17:0/1079526867 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c20005d10 con 0x7f2c3c145e80 2024-03-29T11:21:19.065 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.075+0000 7f2c397fa640 1 -- 172.21.15.17:0/1079526867 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f2c34001bd0 con 0x7f2c3c145e80 2024-03-29T11:21:19.065 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.075+0000 7f2c397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:19.065 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.075+0000 7f2c397fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:19.065 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.075+0000 7f2c397fa640 10 monclient(hunting): dump: 2024-03-29T11:21:19.065 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:21:19.065 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:21:19.065 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:19.065 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:19.065 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:19.065 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:19.065 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:21:19.065 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:21:19.066 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:21:19.066 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:19.066 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.075+0000 7f2c397fa640 1 -- 172.21.15.17:0/1079526867 <== mon.0 v1:172.21.15.17:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2093322161 0 0) 0x7f2c20005640 con 0x7f2c3c06cea0 2024-03-29T11:21:19.066 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.075+0000 7f2c397fa640 1 -- 172.21.15.17:0/1079526867 >> v1:172.21.15.17:6790/0 conn(0x7f2c3c145e80 legacy=0x7f2c3c1e1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:19.066 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.075+0000 7f2c397fa640 1 -- 172.21.15.17:0/1079526867 >> v1:172.21.15.94:6789/0 conn(0x7f2c3c06cac0 legacy=0x7f2c3c0ae460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:21:19.066 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.075+0000 7f2c397fa640 10 monclient: _finish_hunting 0 2024-03-29T11:21:19.066 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.075+0000 7f2c397fa640 1 monclient: found mon.a 2024-03-29T11:21:19.066 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.075+0000 7f2c397fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:19.066 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.075+0000 7f2c397fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:21:19.066 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.075+0000 7f2c397fa640 1 -- 172.21.15.17:0/1079526867 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c3c1d7680 con 0x7f2c3c06cea0 2024-03-29T11:21:19.066 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.075+0000 7f2c397fa640 10 monclient: _finish_auth 0 2024-03-29T11:21:19.066 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.075+0000 7f2c397fa640 10 monclient: _check_auth_tickets 2024-03-29T11:21:19.066 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.075+0000 7f2c397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:19.066 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.075+0000 7f2c42f15640 5 monclient: authenticate success, global_id 4314 2024-03-29T11:21:19.067 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.075+0000 7f2c42f15640 10 monclient: _renew_subs 2024-03-29T11:21:19.067 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.075+0000 7f2c42f15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:21:19.067 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.075+0000 7f2c42f15640 1 -- 172.21.15.17:0/1079526867 --> v1:172.21.15.17:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c3c1d82b0 con 0x7f2c3c06cea0 2024-03-29T11:21:19.067 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.075+0000 7f2c42f15640 10 client.4314.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:19.067 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.075+0000 7f2c42f15640 10 monclient: _renew_subs 2024-03-29T11:21:19.067 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.075+0000 7f2c42f15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:21:19.067 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.075+0000 7f2c42f15640 1 -- 172.21.15.17:0/1079526867 --> v1:172.21.15.17:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c3c0aef80 con 0x7f2c3c06cea0 2024-03-29T11:21:19.067 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.075+0000 7f2c42f15640 1 librados: init done 2024-03-29T11:21:19.067 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.076+0000 7f2c397fa640 1 -- 172.21.15.17:0/1079526867 <== mon.0 v1:172.21.15.17:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2c280047e0 con 0x7f2c3c06cea0 2024-03-29T11:21:19.067 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.076+0000 7f2c397fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:19.067 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.076+0000 7f2c397fa640 1 -- 172.21.15.17:0/1079526867 <== mon.0 v1:172.21.15.17:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f2c28001990 con 0x7f2c3c06cea0 2024-03-29T11:21:19.067 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.076+0000 7f2c397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:19.067 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.076+0000 7f2c3a7fc640 4 set_mon_vals no callback set 2024-03-29T11:21:19.067 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.076+0000 7f2c397fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:19.068 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.076+0000 7f2c397fa640 10 monclient: dump: 2024-03-29T11:21:19.068 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:21:19.068 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:21:19.068 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:19.068 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:19.068 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:19.068 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:19.068 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:21:19.068 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:21:19.068 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:21:19.068 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:19.068 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.076+0000 7f2c42f15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:19.068 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.077+0000 7f2c42f15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:19.068 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.077+0000 7f2c42f15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:21:19.068 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.077+0000 7f2c42f15640 1 -- 172.21.15.17:0/1079526867 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c1c005080 con 0x7f2c3c06cea0 2024-03-29T11:21:19.068 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.078+0000 7f2c397fa640 1 -- 172.21.15.17:0/1079526867 <== mon.0 v1:172.21.15.17:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f2c28004a50 con 0x7f2c3c06cea0 2024-03-29T11:21:19.069 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.079+0000 7f2c40cb1640 10 monclient: get_auth_request con 0x7f2c2006e7a0 auth_method 0 2024-03-29T11:21:19.069 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.079+0000 7f2c397fa640 10 client.4314.objecter ms_handle_connect 0x7f2c2006e7a0 2024-03-29T11:21:19.069 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.079+0000 7f2c397fa640 1 -- 172.21.15.17:0/1079526867 <== mon.0 v1:172.21.15.17:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f2c28054510 con 0x7f2c3c06cea0 2024-03-29T11:21:19.069 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.079+0000 7f2c397fa640 10 client.4314.objecter ms_dispatch 0x7f2c3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:19.069 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.079+0000 7f2c397fa640 3 client.4314.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:19.069 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.079+0000 7f2c397fa640 3 client.4314.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:19.069 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.079+0000 7f2c397fa640 20 client.4314.objecter dump_active .. 0 homeless 2024-03-29T11:21:19.070 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.088+0000 7f2c397fa640 1 -- 172.21.15.17:0/1079526867 <== mon.0 v1:172.21.15.17:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2c280507b0 con 0x7f2c3c06cea0 2024-03-29T11:21:19.070 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.088+0000 7f2c397fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:19.070 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.088+0000 7f2c397fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:19.439 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.462+0000 7f2c42f15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:19.440 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.462+0000 7f2c42f15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:19.440 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.462+0000 7f2c42f15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:21:19.440 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.462+0000 7f2c42f15640 1 -- 172.21.15.17:0/1079526867 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2c1c005330 con 0x7f2c3c06cea0 2024-03-29T11:21:19.440 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.464+0000 7f2c397fa640 1 -- 172.21.15.17:0/1079526867 <== mon.0 v1:172.21.15.17:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f2c280507b0 con 0x7f2c3c06cea0 2024-03-29T11:21:19.440 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.464+0000 7f2c397fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:19.440 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.464+0000 7f2c397fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:19.441 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:21:19.441 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:19.443 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.467+0000 7f2c42f15640 10 librados: watch_flush enter 2024-03-29T11:21:19.444 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.467+0000 7f2c42f15640 10 librados: watch_flush exit 2024-03-29T11:21:19.444 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.467+0000 7f2c42f15640 20 client.4314.objecter shutdown clearing up homeless session... 2024-03-29T11:21:19.444 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.467+0000 7f2c42f15640 10 client.4314.objecter successfully canceled tick 2024-03-29T11:21:19.444 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.467+0000 7f2c42f15640 1 -- 172.21.15.17:0/1079526867 >> v1:172.21.15.17:6816/3236185052 conn(0x7f2c2006e7a0 legacy=0x7f2c20070c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:19.444 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.467+0000 7f2c42f15640 10 monclient: shutdown 2024-03-29T11:21:19.445 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.467+0000 7f2c42f15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:19.445 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.467+0000 7f2c42f15640 1 -- 172.21.15.17:0/1079526867 >> v1:172.21.15.17:6789/0 conn(0x7f2c3c06cea0 legacy=0x7f2c3c1d7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:19.445 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.468+0000 7f2c42f15640 1 -- 172.21.15.17:0/1079526867 shutdown_connections 2024-03-29T11:21:19.445 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.468+0000 7f2c42f15640 1 -- 172.21.15.17:0/1079526867 >> 172.21.15.17:0/1079526867 conn(0x7f2c3c0675b0 msgr2=0x7f2c3c0b2500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:19.445 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.468+0000 7f2c42f15640 1 -- 172.21.15.17:0/1079526867 shutdown_connections 2024-03-29T11:21:19.445 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.468+0000 7f2c42f15640 1 -- 172.21.15.17:0/1079526867 wait complete. 2024-03-29T11:21:19.445 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:19.468+0000 7f2c42f15640 1 librados: shutdown 2024-03-29T11:21:19.459 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:26.267 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:26.422 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.445+0000 7ff85d39a640 10 monclient: get_monmap_and_config 2024-03-29T11:21:26.422 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.445+0000 7ff85d39a640 10 monclient: build_initial_monmap 2024-03-29T11:21:26.422 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.445+0000 7ff85d39a640 10 monclient: monmap: 2024-03-29T11:21:26.422 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:21:26.422 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:21:26.422 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:21:26.446539+0000 2024-03-29T11:21:26.422 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:21:26.446539+0000 2024-03-29T11:21:26.422 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:26.423 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:26.423 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:21:26.423 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:21:26.423 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:21:26.423 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:26.424 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.447+0000 7ff85d39a640 1 Processor -- start 2024-03-29T11:21:26.424 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.448+0000 7ff85d39a640 1 -- start start 2024-03-29T11:21:26.424 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.448+0000 7ff85d39a640 10 monclient: init 2024-03-29T11:21:26.425 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.449+0000 7ff85d39a640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:26.425 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.449+0000 7ff85d39a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:21:26.425 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.449+0000 7ff85d39a640 10 monclient(hunting): picked mon.noname-c con 0x7ff8581422b0 addr v1:172.21.15.94:6789/0 2024-03-29T11:21:26.426 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.449+0000 7ff85d39a640 10 monclient(hunting): picked mon.noname-a con 0x7ff858142690 addr v1:172.21.15.17:6789/0 2024-03-29T11:21:26.427 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.449+0000 7ff85d39a640 10 monclient(hunting): picked mon.noname-b con 0x7ff8580b17d0 addr v1:172.21.15.17:6790/0 2024-03-29T11:21:26.427 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.450+0000 7ff85d39a640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff858142c80 con 0x7ff858142690 2024-03-29T11:21:26.427 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.450+0000 7ff85d39a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8580aae40 con 0x7ff8581422b0 2024-03-29T11:21:26.427 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.450+0000 7ff85d39a640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff85813fbd0 con 0x7ff8580b17d0 2024-03-29T11:21:26.427 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.450+0000 7ff85d39a640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:26.427 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.450+0000 7ff85d39a640 10 monclient(hunting): authenticate will time out at 992.655222s 2024-03-29T11:21:26.427 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.450+0000 7ff8577fe640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7ff8580b17d0 0x7ff8580b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:59336/0 (socket says 172.21.15.17:59336) 2024-03-29T11:21:26.427 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.450+0000 7ff8577fe640 1 -- 172.21.15.17:0/867425947 learned_addr learned my addr 172.21.15.17:0/867425947 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:21:26.428 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.452+0000 7ff855ffb640 1 -- 172.21.15.17:0/867425947 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2009083343 0 0) 0x7ff85813fbd0 con 0x7ff8580b17d0 2024-03-29T11:21:26.428 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.452+0000 7ff855ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:26.428 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.452+0000 7ff855ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:26.429 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.452+0000 7ff855ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:26.429 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.452+0000 7ff855ffb640 10 monclient(hunting): my global_id is 4328 2024-03-29T11:21:26.429 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.452+0000 7ff855ffb640 1 -- 172.21.15.17:0/867425947 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff840003520 con 0x7ff8580b17d0 2024-03-29T11:21:26.429 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.452+0000 7ff855ffb640 1 -- 172.21.15.17:0/867425947 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 205301970 0 0) 0x7ff840003520 con 0x7ff8580b17d0 2024-03-29T11:21:26.429 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.452+0000 7ff855ffb640 1 -- 172.21.15.17:0/867425947 >> v1:172.21.15.94:6789/0 conn(0x7ff8581422b0 legacy=0x7ff85813f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:21:26.432 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.454+0000 7ff855ffb640 1 -- 172.21.15.17:0/867425947 >> v1:172.21.15.17:6789/0 conn(0x7ff858142690 legacy=0x7ff8580b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:26.432 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.454+0000 7ff855ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:21:26.432 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.454+0000 7ff855ffb640 1 monclient: found mon.noname-b 2024-03-29T11:21:26.432 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.454+0000 7ff855ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:26.432 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.454+0000 7ff855ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.17:6790/0 2024-03-29T11:21:26.432 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.454+0000 7ff855ffb640 1 -- 172.21.15.17:0/867425947 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff858151940 con 0x7ff8580b17d0 2024-03-29T11:21:26.433 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.454+0000 7ff855ffb640 10 monclient: _finish_auth 0 2024-03-29T11:21:26.433 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.454+0000 7ff855ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:21:26.433 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.454+0000 7ff855ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:26.433 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.454+0000 7ff855ffb640 1 -- 172.21.15.17:0/867425947 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7ff850001a10 con 0x7ff8580b17d0 2024-03-29T11:21:26.433 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.455+0000 7ff85d39a640 5 monclient: authenticate success, global_id 4328 2024-03-29T11:21:26.433 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.455+0000 7ff85d39a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:26.433 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.455+0000 7ff855ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:26.433 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.455+0000 7ff855ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:26.433 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.455+0000 7ff855ffb640 10 monclient: dump: 2024-03-29T11:21:26.433 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:21:26.433 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:21:26.433 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:26.433 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:26.433 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:26.434 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:26.434 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:21:26.434 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:21:26.434 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:21:26.434 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:26.434 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.455+0000 7ff855ffb640 1 -- 172.21.15.17:0/867425947 <== mon.2 v1:172.21.15.17:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff850001c30 con 0x7ff8580b17d0 2024-03-29T11:21:26.434 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.455+0000 7ff855ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:26.434 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.455+0000 7ff855ffb640 1 -- 172.21.15.17:0/867425947 <== mon.2 v1:172.21.15.17:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7ff850001fc0 con 0x7ff8580b17d0 2024-03-29T11:21:26.434 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.455+0000 7ff855ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:26.434 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.455+0000 7ff855ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:26.434 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.455+0000 7ff855ffb640 10 monclient: dump: 2024-03-29T11:21:26.434 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:21:26.434 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:21:26.434 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:26.435 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:26.435 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:26.435 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:26.435 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:21:26.435 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:21:26.435 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:21:26.435 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:26.435 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.455+0000 7ff85d39a640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:26.435 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.455+0000 7ff85d39a640 4 set_mon_vals no callback set 2024-03-29T11:21:26.436 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.456+0000 7ff85d39a640 10 monclient: shutdown 2024-03-29T11:21:26.436 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.457+0000 7ff85d39a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:26.437 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.457+0000 7ff85d39a640 1 -- 172.21.15.17:0/867425947 >> v1:172.21.15.17:6790/0 conn(0x7ff8580b17d0 legacy=0x7ff8580b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:26.437 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.457+0000 7ff85d39a640 1 -- 172.21.15.17:0/867425947 shutdown_connections 2024-03-29T11:21:26.437 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.457+0000 7ff85d39a640 1 -- 172.21.15.17:0/867425947 >> 172.21.15.17:0/867425947 conn(0x7ff858066a00 msgr2=0x7ff858066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:26.437 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.458+0000 7ff85d39a640 1 -- 172.21.15.17:0/867425947 shutdown_connections 2024-03-29T11:21:26.437 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.459+0000 7ff85d39a640 1 -- 172.21.15.17:0/867425947 wait complete. 2024-03-29T11:21:26.437 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.460+0000 7ff85d39a640 10 monclient: build_initial_monmap 2024-03-29T11:21:26.437 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.460+0000 7ff85d39a640 10 monclient: monmap: 2024-03-29T11:21:26.437 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:21:26.437 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:26.437 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:21:26.437 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:21:26.437 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:26.437 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:26.437 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:21:26.438 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:21:26.438 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:21:26.438 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:26.438 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.460+0000 7ff85d39a640 1 librados: starting msgr at 2024-03-29T11:21:26.438 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.460+0000 7ff85d39a640 1 librados: starting objecter 2024-03-29T11:21:26.438 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.460+0000 7ff85d39a640 1 Processor -- start 2024-03-29T11:21:26.438 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.461+0000 7ff85d39a640 1 -- start start 2024-03-29T11:21:26.439 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.461+0000 7ff85d39a640 1 librados: setting wanted keys 2024-03-29T11:21:26.439 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.461+0000 7ff85d39a640 1 librados: calling monclient init 2024-03-29T11:21:26.439 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.461+0000 7ff85d39a640 10 monclient: init 2024-03-29T11:21:26.439 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.461+0000 7ff85d39a640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:26.439 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.462+0000 7ff85d39a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:21:26.440 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.462+0000 7ff85d39a640 10 monclient(hunting): picked mon.noname-b con 0x7ff8580b17d0 addr v1:172.21.15.94:6789/0 2024-03-29T11:21:26.441 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.464+0000 7ff85d39a640 10 monclient(hunting): picked mon.noname-a con 0x7ff8581422b0 addr v1:172.21.15.17:6789/0 2024-03-29T11:21:26.441 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.464+0000 7ff85d39a640 10 monclient(hunting): picked mon.noname-c con 0x7ff858142690 addr v1:172.21.15.17:6790/0 2024-03-29T11:21:26.441 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.464+0000 7ff85d39a640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8580acaf0 con 0x7ff8581422b0 2024-03-29T11:21:26.441 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.464+0000 7ff85d39a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff858071a60 con 0x7ff8580b17d0 2024-03-29T11:21:26.441 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.464+0000 7ff85d39a640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8580acd60 con 0x7ff858142690 2024-03-29T11:21:26.441 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.464+0000 7ff85d39a640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:26.441 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.464+0000 7ff85d39a640 10 monclient(hunting): authenticate will time out at 992.669049s 2024-03-29T11:21:26.451 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.466+0000 7ff8577fe640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7ff858142690 0x7ff8581e4420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:60636/0 (socket says 172.21.15.17:60636) 2024-03-29T11:21:26.451 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.466+0000 7ff8577fe640 1 -- 172.21.15.17:0/3499614275 learned_addr learned my addr 172.21.15.17:0/3499614275 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:21:26.451 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.466+0000 7ff83f7fe640 10 client.?.objecter ms_handle_connect 0x7ff858142690 2024-03-29T11:21:26.452 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.466+0000 7ff83f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:26.452 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.466+0000 7ff83f7fe640 10 client.?.objecter ms_handle_connect 0x7ff8581422b0 2024-03-29T11:21:26.452 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.466+0000 7ff83f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:26.452 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.466+0000 7ff83f7fe640 1 -- 172.21.15.17:0/3499614275 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 857545876 0 0) 0x7ff8580acd60 con 0x7ff858142690 2024-03-29T11:21:26.452 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.467+0000 7ff83f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:26.452 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.467+0000 7ff83f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:26.452 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.467+0000 7ff83f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:26.452 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.467+0000 7ff83f7fe640 10 monclient(hunting): my global_id is 4331 2024-03-29T11:21:26.452 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.467+0000 7ff83f7fe640 1 -- 172.21.15.17:0/3499614275 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff844004eb0 con 0x7ff858142690 2024-03-29T11:21:26.452 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.467+0000 7ff83f7fe640 10 client.?.objecter ms_handle_connect 0x7ff8580b17d0 2024-03-29T11:21:26.452 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.467+0000 7ff83f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:26.452 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.467+0000 7ff83f7fe640 1 -- 172.21.15.17:0/3499614275 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1038626280 0 0) 0x7ff8580acaf0 con 0x7ff8581422b0 2024-03-29T11:21:26.452 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.467+0000 7ff83f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:26.452 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.467+0000 7ff83f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:26.453 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.467+0000 7ff83f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:26.453 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.467+0000 7ff83f7fe640 10 monclient(hunting): my global_id is 4320 2024-03-29T11:21:26.453 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.467+0000 7ff83f7fe640 1 -- 172.21.15.17:0/3499614275 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8580acd60 con 0x7ff8581422b0 2024-03-29T11:21:26.453 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.467+0000 7ff83f7fe640 1 -- 172.21.15.17:0/3499614275 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3389564033 0 0) 0x7ff844004eb0 con 0x7ff858142690 2024-03-29T11:21:26.453 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.467+0000 7ff83f7fe640 1 -- 172.21.15.17:0/3499614275 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8580acaf0 con 0x7ff858142690 2024-03-29T11:21:26.453 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.467+0000 7ff83f7fe640 1 -- 172.21.15.17:0/3499614275 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 673064022 0 0) 0x7ff8580acd60 con 0x7ff8581422b0 2024-03-29T11:21:26.453 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.467+0000 7ff83f7fe640 1 -- 172.21.15.17:0/3499614275 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff844004eb0 con 0x7ff8581422b0 2024-03-29T11:21:26.453 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.467+0000 7ff83f7fe640 1 -- 172.21.15.17:0/3499614275 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7ff8500018d0 con 0x7ff858142690 2024-03-29T11:21:26.453 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.467+0000 7ff83f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:26.453 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.467+0000 7ff83f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:26.453 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.467+0000 7ff83f7fe640 10 monclient(hunting): dump: 2024-03-29T11:21:26.453 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:21:26.453 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:21:26.453 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:26.454 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:26.454 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:26.454 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:26.454 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:21:26.454 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:21:26.454 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:21:26.454 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:26.454 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.467+0000 7ff83f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:26.454 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.467+0000 7ff83f7fe640 1 -- 172.21.15.17:0/3499614275 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7ff84c001810 con 0x7ff8581422b0 2024-03-29T11:21:26.454 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.467+0000 7ff83f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:26.454 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.467+0000 7ff83f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:26.454 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.467+0000 7ff83f7fe640 10 monclient(hunting): dump: 2024-03-29T11:21:26.454 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:21:26.454 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:21:26.455 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:26.455 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:26.455 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:26.455 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:26.455 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:21:26.455 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:21:26.455 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:21:26.455 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:26.455 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.468+0000 7ff83f7fe640 1 -- 172.21.15.17:0/3499614275 <== mon.2 v1:172.21.15.17:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 39828272 0 0) 0x7ff8580acaf0 con 0x7ff858142690 2024-03-29T11:21:26.455 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.468+0000 7ff83f7fe640 1 -- 172.21.15.17:0/3499614275 >> v1:172.21.15.94:6789/0 conn(0x7ff8580b17d0 legacy=0x7ff8580ab7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:26.455 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.468+0000 7ff83f7fe640 1 -- 172.21.15.17:0/3499614275 >> v1:172.21.15.17:6789/0 conn(0x7ff8581422b0 legacy=0x7ff85813bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:26.455 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.468+0000 7ff83f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:21:26.455 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.468+0000 7ff83f7fe640 1 monclient: found mon.c 2024-03-29T11:21:26.455 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.468+0000 7ff83f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:26.456 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.468+0000 7ff83f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:21:26.456 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.468+0000 7ff83f7fe640 1 -- 172.21.15.17:0/3499614275 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8580ad040 con 0x7ff858142690 2024-03-29T11:21:26.456 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.468+0000 7ff83f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:21:26.456 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.468+0000 7ff83f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:21:26.456 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.468+0000 7ff83f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:26.456 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.468+0000 7ff85d39a640 5 monclient: authenticate success, global_id 4331 2024-03-29T11:21:26.456 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.469+0000 7ff85d39a640 10 monclient: _renew_subs 2024-03-29T11:21:26.456 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.469+0000 7ff85d39a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:21:26.456 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.469+0000 7ff85d39a640 1 -- 172.21.15.17:0/3499614275 --> v1:172.21.15.17:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8580716f0 con 0x7ff858142690 2024-03-29T11:21:26.456 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.469+0000 7ff85d39a640 10 client.4331.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:26.457 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.469+0000 7ff85d39a640 10 monclient: _renew_subs 2024-03-29T11:21:26.457 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.469+0000 7ff83f7fe640 1 -- 172.21.15.17:0/3499614275 <== mon.2 v1:172.21.15.17:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff850001a30 con 0x7ff858142690 2024-03-29T11:21:26.457 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.469+0000 7ff85d39a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:21:26.457 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.469+0000 7ff85d39a640 1 -- 172.21.15.17:0/3499614275 --> v1:172.21.15.17:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff85813c670 con 0x7ff858142690 2024-03-29T11:21:26.457 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.469+0000 7ff83f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:26.457 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.469+0000 7ff83f7fe640 1 -- 172.21.15.17:0/3499614275 <== mon.2 v1:172.21.15.17:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7ff850005110 con 0x7ff858142690 2024-03-29T11:21:26.457 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.469+0000 7ff83f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:26.457 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.469+0000 7ff83f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:26.457 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.469+0000 7ff83f7fe640 10 monclient: dump: 2024-03-29T11:21:26.457 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:21:26.457 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:21:26.457 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:26.457 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:26.457 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:26.457 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:26.458 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:21:26.458 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:21:26.458 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:21:26.458 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:26.458 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.469+0000 7ff85d39a640 1 librados: init done 2024-03-29T11:21:26.458 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.470+0000 7ff854ff9640 4 set_mon_vals no callback set 2024-03-29T11:21:26.458 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.471+0000 7ff83f7fe640 1 -- 172.21.15.17:0/3499614275 <== mon.2 v1:172.21.15.17:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7ff85001b0e0 con 0x7ff858142690 2024-03-29T11:21:26.458 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.472+0000 7ff85d39a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:26.458 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.472+0000 7ff83f7fe640 1 -- 172.21.15.17:0/3499614275 <== mon.2 v1:172.21.15.17:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7ff850084ed0 con 0x7ff858142690 2024-03-29T11:21:26.458 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.472+0000 7ff83f7fe640 10 client.4331.objecter ms_dispatch 0x7ff858000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:26.458 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.472+0000 7ff83f7fe640 3 client.4331.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:26.458 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.472+0000 7ff83f7fe640 3 client.4331.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:26.458 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.472+0000 7ff83f7fe640 20 client.4331.objecter dump_active .. 0 homeless 2024-03-29T11:21:26.459 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.472+0000 7ff85d39a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:26.459 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.472+0000 7ff85d39a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:21:26.459 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.472+0000 7ff85d39a640 1 -- 172.21.15.17:0/3499614275 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8580abec0 con 0x7ff858142690 2024-03-29T11:21:26.459 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.472+0000 7ff856ffd640 10 monclient: get_auth_request con 0x7ff84406dec0 auth_method 0 2024-03-29T11:21:26.459 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.473+0000 7ff83f7fe640 10 client.4331.objecter ms_handle_connect 0x7ff84406dec0 2024-03-29T11:21:26.460 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.481+0000 7ff83f7fe640 1 -- 172.21.15.17:0/3499614275 <== mon.2 v1:172.21.15.17:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff8500508f0 con 0x7ff858142690 2024-03-29T11:21:26.460 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.481+0000 7ff83f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:26.460 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.481+0000 7ff83f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:26.789 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.812+0000 7ff85d39a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:26.789 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.812+0000 7ff85d39a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:26.789 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.812+0000 7ff85d39a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:21:26.790 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.812+0000 7ff85d39a640 1 -- 172.21.15.17:0/3499614275 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff858075f10 con 0x7ff858142690 2024-03-29T11:21:26.790 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.813+0000 7ff83f7fe640 1 -- 172.21.15.17:0/3499614275 <== mon.2 v1:172.21.15.17:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7ff850052820 con 0x7ff858142690 2024-03-29T11:21:26.790 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.813+0000 7ff83f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:26.790 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.813+0000 7ff83f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:26.793 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:21:26.793 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:26.793 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.816+0000 7ff85d39a640 10 librados: watch_flush enter 2024-03-29T11:21:26.794 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.817+0000 7ff85d39a640 10 librados: watch_flush exit 2024-03-29T11:21:26.794 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.817+0000 7ff85d39a640 20 client.4331.objecter shutdown clearing up homeless session... 2024-03-29T11:21:26.794 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.817+0000 7ff85d39a640 10 client.4331.objecter successfully canceled tick 2024-03-29T11:21:26.794 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.817+0000 7ff85d39a640 1 -- 172.21.15.17:0/3499614275 >> v1:172.21.15.17:6816/3236185052 conn(0x7ff84406dec0 legacy=0x7ff844070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:26.794 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.817+0000 7ff85d39a640 10 monclient: shutdown 2024-03-29T11:21:26.794 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.817+0000 7ff85d39a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:26.794 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.817+0000 7ff85d39a640 1 -- 172.21.15.17:0/3499614275 >> v1:172.21.15.17:6790/0 conn(0x7ff858142690 legacy=0x7ff8581e4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:26.795 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.818+0000 7ff85d39a640 1 -- 172.21.15.17:0/3499614275 shutdown_connections 2024-03-29T11:21:26.795 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.818+0000 7ff85d39a640 1 -- 172.21.15.17:0/3499614275 >> 172.21.15.17:0/3499614275 conn(0x7ff858066a00 msgr2=0x7ff858141660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:26.795 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.818+0000 7ff85d39a640 1 -- 172.21.15.17:0/3499614275 shutdown_connections 2024-03-29T11:21:26.795 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.818+0000 7ff85d39a640 1 -- 172.21.15.17:0/3499614275 wait complete. 2024-03-29T11:21:26.796 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:26.819+0000 7ff85d39a640 1 librados: shutdown 2024-03-29T11:21:26.813 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:33.620 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:33.772 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.793+0000 7f2c921e3640 10 monclient: get_monmap_and_config 2024-03-29T11:21:33.772 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.793+0000 7f2c921e3640 10 monclient: build_initial_monmap 2024-03-29T11:21:33.772 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.793+0000 7f2c921e3640 10 monclient: monmap: 2024-03-29T11:21:33.772 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:21:33.772 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:21:33.772 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:21:33.794473+0000 2024-03-29T11:21:33.772 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:21:33.794473+0000 2024-03-29T11:21:33.772 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:33.773 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:33.773 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:21:33.773 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:21:33.773 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:21:33.773 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:33.774 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.794+0000 7f2c921e3640 1 Processor -- start 2024-03-29T11:21:33.774 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.795+0000 7f2c921e3640 1 -- start start 2024-03-29T11:21:33.774 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.795+0000 7f2c921e3640 10 monclient: init 2024-03-29T11:21:33.774 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.795+0000 7f2c921e3640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:33.774 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.795+0000 7f2c921e3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:21:33.774 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.797+0000 7f2c921e3640 10 monclient(hunting): picked mon.noname-b con 0x7f2c8402b090 addr v1:172.21.15.17:6790/0 2024-03-29T11:21:33.774 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.797+0000 7f2c921e3640 10 monclient(hunting): picked mon.noname-a con 0x7f2c84006940 addr v1:172.21.15.17:6789/0 2024-03-29T11:21:33.774 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.797+0000 7f2c911e1640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f2c8402b090 0x7f2c840e8900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:35966/0 (socket says 172.21.15.17:35966) 2024-03-29T11:21:33.774 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.797+0000 7f2c921e3640 10 monclient(hunting): picked mon.noname-c con 0x7f2c840e2530 addr v1:172.21.15.94:6789/0 2024-03-29T11:21:33.774 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.797+0000 7f2c921e3640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c84006d80 con 0x7f2c84006940 2024-03-29T11:21:33.775 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.797+0000 7f2c921e3640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c8402a430 con 0x7f2c840e2530 2024-03-29T11:21:33.775 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.797+0000 7f2c921e3640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c8402a5c0 con 0x7f2c8402b090 2024-03-29T11:21:33.775 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.797+0000 7f2c921e3640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:33.775 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.797+0000 7f2c921e3640 10 monclient(hunting): authenticate will time out at 1000.002623s 2024-03-29T11:21:33.775 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.797+0000 7f2c911e1640 1 -- 172.21.15.17:0/2709204082 learned_addr learned my addr 172.21.15.17:0/2709204082 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:21:33.778 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.801+0000 7f2c83fff640 1 -- 172.21.15.17:0/2709204082 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1244289533 0 0) 0x7f2c8402a5c0 con 0x7f2c8402b090 2024-03-29T11:21:33.778 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.801+0000 7f2c83fff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:33.779 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.801+0000 7f2c83fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:33.779 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.801+0000 7f2c83fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:33.779 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.801+0000 7f2c83fff640 10 monclient(hunting): my global_id is 4334 2024-03-29T11:21:33.779 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.801+0000 7f2c83fff640 1 -- 172.21.15.17:0/2709204082 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c78003520 con 0x7f2c8402b090 2024-03-29T11:21:33.779 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.801+0000 7f2c83fff640 1 -- 172.21.15.17:0/2709204082 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2677559154 0 0) 0x7f2c84006d80 con 0x7f2c84006940 2024-03-29T11:21:33.779 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.801+0000 7f2c83fff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:33.779 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.801+0000 7f2c83fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:33.779 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.801+0000 7f2c83fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:33.779 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.801+0000 7f2c83fff640 10 monclient(hunting): my global_id is 4323 2024-03-29T11:21:33.779 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.801+0000 7f2c83fff640 1 -- 172.21.15.17:0/2709204082 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c8402a5c0 con 0x7f2c84006940 2024-03-29T11:21:33.779 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.801+0000 7f2c83fff640 1 -- 172.21.15.17:0/2709204082 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 896067560 0 0) 0x7f2c8402a5c0 con 0x7f2c84006940 2024-03-29T11:21:33.779 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.801+0000 7f2c83fff640 1 -- 172.21.15.17:0/2709204082 >> v1:172.21.15.17:6790/0 conn(0x7f2c8402b090 legacy=0x7f2c840e8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:33.779 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.801+0000 7f2c83fff640 1 -- 172.21.15.17:0/2709204082 >> v1:172.21.15.94:6789/0 conn(0x7f2c840e2530 legacy=0x7f2c840e29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:33.780 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.801+0000 7f2c83fff640 10 monclient: _finish_hunting 0 2024-03-29T11:21:33.780 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.801+0000 7f2c83fff640 1 monclient: found mon.noname-a 2024-03-29T11:21:33.780 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.801+0000 7f2c83fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:33.780 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.801+0000 7f2c83fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.17:6789/0 2024-03-29T11:21:33.780 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.801+0000 7f2c83fff640 1 -- 172.21.15.17:0/2709204082 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c840eeda0 con 0x7f2c84006940 2024-03-29T11:21:33.780 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.801+0000 7f2c83fff640 10 monclient: _finish_auth 0 2024-03-29T11:21:33.780 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.801+0000 7f2c83fff640 10 monclient: _check_auth_tickets 2024-03-29T11:21:33.780 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.801+0000 7f2c83fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:33.780 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.801+0000 7f2c83fff640 1 -- 172.21.15.17:0/2709204082 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f2c8c04f730 con 0x7f2c84006940 2024-03-29T11:21:33.780 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.801+0000 7f2c921e3640 5 monclient: authenticate success, global_id 4323 2024-03-29T11:21:33.780 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.801+0000 7f2c921e3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:33.780 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.801+0000 7f2c83fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:33.780 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.801+0000 7f2c83fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:33.780 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.801+0000 7f2c83fff640 10 monclient: dump: 2024-03-29T11:21:33.780 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:21:33.781 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:21:33.781 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:33.781 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:33.781 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:33.781 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:33.781 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:21:33.781 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:21:33.781 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:21:33.781 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:33.781 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.801+0000 7f2c921e3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:33.781 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.801+0000 7f2c83fff640 1 -- 172.21.15.17:0/2709204082 <== mon.0 v1:172.21.15.17:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2c8c0008d0 con 0x7f2c84006940 2024-03-29T11:21:33.781 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.801+0000 7f2c83fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:33.781 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.801+0000 7f2c83fff640 1 -- 172.21.15.17:0/2709204082 <== mon.0 v1:172.21.15.17:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f2c8c0758b0 con 0x7f2c84006940 2024-03-29T11:21:33.781 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.801+0000 7f2c83fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:33.781 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.801+0000 7f2c83fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:33.782 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.801+0000 7f2c83fff640 10 monclient: dump: 2024-03-29T11:21:33.782 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:21:33.782 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:21:33.782 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:33.782 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:33.782 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:33.782 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:33.782 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:21:33.782 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:21:33.782 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:21:33.782 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:33.782 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.802+0000 7f2c921e3640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:33.782 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.802+0000 7f2c921e3640 4 set_mon_vals no callback set 2024-03-29T11:21:33.784 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.802+0000 7f2c921e3640 10 monclient: shutdown 2024-03-29T11:21:33.784 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.803+0000 7f2c921e3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:33.784 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.803+0000 7f2c921e3640 1 -- 172.21.15.17:0/2709204082 >> v1:172.21.15.17:6789/0 conn(0x7f2c84006940 legacy=0x7f2c84029c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:33.784 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.803+0000 7f2c921e3640 1 -- 172.21.15.17:0/2709204082 shutdown_connections 2024-03-29T11:21:33.784 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.803+0000 7f2c921e3640 1 -- 172.21.15.17:0/2709204082 >> 172.21.15.17:0/2709204082 conn(0x7f2c840058f0 msgr2=0x7f2c84005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:33.784 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.805+0000 7f2c921e3640 1 -- 172.21.15.17:0/2709204082 shutdown_connections 2024-03-29T11:21:33.784 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.805+0000 7f2c921e3640 1 -- 172.21.15.17:0/2709204082 wait complete. 2024-03-29T11:21:33.784 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.806+0000 7f2c921e3640 10 monclient: build_initial_monmap 2024-03-29T11:21:33.784 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.806+0000 7f2c921e3640 10 monclient: monmap: 2024-03-29T11:21:33.784 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:21:33.784 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:33.785 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:21:33.785 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:21:33.785 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:33.785 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:33.785 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:21:33.785 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:21:33.785 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:21:33.785 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:33.785 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.806+0000 7f2c921e3640 1 librados: starting msgr at 2024-03-29T11:21:33.785 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.806+0000 7f2c921e3640 1 librados: starting objecter 2024-03-29T11:21:33.785 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.806+0000 7f2c921e3640 1 Processor -- start 2024-03-29T11:21:33.785 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.807+0000 7f2c921e3640 1 -- start start 2024-03-29T11:21:33.785 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.807+0000 7f2c921e3640 1 librados: setting wanted keys 2024-03-29T11:21:33.785 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.807+0000 7f2c921e3640 1 librados: calling monclient init 2024-03-29T11:21:33.786 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.807+0000 7f2c921e3640 10 monclient: init 2024-03-29T11:21:33.787 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.808+0000 7f2c921e3640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:33.787 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.808+0000 7f2c921e3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:21:33.787 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.808+0000 7f2c921e3640 10 monclient(hunting): picked mon.noname-a con 0x7f2c84006940 addr v1:172.21.15.17:6789/0 2024-03-29T11:21:33.787 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.808+0000 7f2c921e3640 10 monclient(hunting): picked mon.noname-b con 0x7f2c8402b090 addr v1:172.21.15.94:6789/0 2024-03-29T11:21:33.787 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.808+0000 7f2c921e3640 10 monclient(hunting): picked mon.noname-c con 0x7f2c840e2530 addr v1:172.21.15.17:6790/0 2024-03-29T11:21:33.787 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.808+0000 7f2c921e3640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c840dfa30 con 0x7f2c84006940 2024-03-29T11:21:33.788 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.808+0000 7f2c921e3640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c840ef030 con 0x7f2c8402b090 2024-03-29T11:21:33.788 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.808+0000 7f2c921e3640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c840de2c0 con 0x7f2c840e2530 2024-03-29T11:21:33.788 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.808+0000 7f2c921e3640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:33.788 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.808+0000 7f2c921e3640 10 monclient(hunting): authenticate will time out at 1000.013322s 2024-03-29T11:21:33.788 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.811+0000 7f2c919e2640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f2c840e2530 0x7f2c84189880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:39630/0 (socket says 172.21.15.17:39630) 2024-03-29T11:21:33.793 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.811+0000 7f2c919e2640 1 -- 172.21.15.17:0/487192413 learned_addr learned my addr 172.21.15.17:0/487192413 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:21:33.793 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.812+0000 7f2c81ffb640 10 client.?.objecter ms_handle_connect 0x7f2c84006940 2024-03-29T11:21:33.793 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.812+0000 7f2c81ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:33.793 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.812+0000 7f2c81ffb640 10 client.?.objecter ms_handle_connect 0x7f2c840e2530 2024-03-29T11:21:33.793 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.812+0000 7f2c81ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:33.793 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.812+0000 7f2c81ffb640 1 -- 172.21.15.17:0/487192413 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1040567197 0 0) 0x7f2c840de2c0 con 0x7f2c840e2530 2024-03-29T11:21:33.793 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.812+0000 7f2c81ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:33.793 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.812+0000 7f2c81ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:33.793 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.812+0000 7f2c81ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:33.793 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.812+0000 7f2c81ffb640 10 monclient(hunting): my global_id is 4337 2024-03-29T11:21:33.794 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.812+0000 7f2c81ffb640 1 -- 172.21.15.17:0/487192413 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c70004eb0 con 0x7f2c840e2530 2024-03-29T11:21:33.794 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.812+0000 7f2c81ffb640 1 -- 172.21.15.17:0/487192413 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3355821914 0 0) 0x7f2c840dfa30 con 0x7f2c84006940 2024-03-29T11:21:33.794 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.812+0000 7f2c81ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:33.794 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.812+0000 7f2c81ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:33.794 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.812+0000 7f2c81ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:33.794 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.812+0000 7f2c81ffb640 10 monclient(hunting): my global_id is 4326 2024-03-29T11:21:33.794 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.812+0000 7f2c81ffb640 1 -- 172.21.15.17:0/487192413 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c840de2c0 con 0x7f2c84006940 2024-03-29T11:21:33.794 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.813+0000 7f2c81ffb640 1 -- 172.21.15.17:0/487192413 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3311186493 0 0) 0x7f2c70004eb0 con 0x7f2c840e2530 2024-03-29T11:21:33.794 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.813+0000 7f2c81ffb640 1 -- 172.21.15.17:0/487192413 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c840dfa30 con 0x7f2c840e2530 2024-03-29T11:21:33.794 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.813+0000 7f2c81ffb640 1 -- 172.21.15.17:0/487192413 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f2c7c003f60 con 0x7f2c840e2530 2024-03-29T11:21:33.794 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.813+0000 7f2c81ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:33.794 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.813+0000 7f2c81ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:33.794 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.813+0000 7f2c81ffb640 10 monclient(hunting): dump: 2024-03-29T11:21:33.794 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:21:33.795 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:21:33.795 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:33.795 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:33.795 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:33.795 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:33.795 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:21:33.795 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:21:33.795 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:21:33.795 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:33.795 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.813+0000 7f2c81ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:33.795 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.813+0000 7f2c81ffb640 1 -- 172.21.15.17:0/487192413 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3924600466 0 0) 0x7f2c840de2c0 con 0x7f2c84006940 2024-03-29T11:21:33.795 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.813+0000 7f2c81ffb640 1 -- 172.21.15.17:0/487192413 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c70004eb0 con 0x7f2c84006940 2024-03-29T11:21:33.795 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.813+0000 7f2c81ffb640 1 -- 172.21.15.17:0/487192413 <== mon.2 v1:172.21.15.17:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1015289872 0 0) 0x7f2c840dfa30 con 0x7f2c840e2530 2024-03-29T11:21:33.795 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.813+0000 7f2c81ffb640 1 -- 172.21.15.17:0/487192413 >> v1:172.21.15.94:6789/0 conn(0x7f2c8402b090 legacy=0x7f2c84186150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:33.796 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.814+0000 7f2c81ffb640 1 -- 172.21.15.17:0/487192413 >> v1:172.21.15.17:6789/0 conn(0x7f2c84006940 legacy=0x7f2c840ef450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:33.796 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.814+0000 7f2c81ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:21:33.796 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.814+0000 7f2c81ffb640 1 monclient: found mon.c 2024-03-29T11:21:33.796 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.814+0000 7f2c81ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:33.796 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.814+0000 7f2c81ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:21:33.796 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.814+0000 7f2c81ffb640 1 -- 172.21.15.17:0/487192413 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c840efb60 con 0x7f2c840e2530 2024-03-29T11:21:33.796 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.815+0000 7f2c81ffb640 10 monclient: _finish_auth 0 2024-03-29T11:21:33.796 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.815+0000 7f2c81ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:21:33.796 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.815+0000 7f2c81ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:33.796 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.815+0000 7f2c921e3640 5 monclient: authenticate success, global_id 4337 2024-03-29T11:21:33.796 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.815+0000 7f2c921e3640 10 monclient: _renew_subs 2024-03-29T11:21:33.796 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.815+0000 7f2c921e3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:21:33.796 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.815+0000 7f2c921e3640 1 -- 172.21.15.17:0/487192413 --> v1:172.21.15.17:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c840dc740 con 0x7f2c840e2530 2024-03-29T11:21:33.796 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.815+0000 7f2c81ffb640 1 -- 172.21.15.17:0/487192413 <== mon.2 v1:172.21.15.17:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2c7c0040c0 con 0x7f2c840e2530 2024-03-29T11:21:33.796 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.815+0000 7f2c81ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:33.797 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.815+0000 7f2c81ffb640 1 -- 172.21.15.17:0/487192413 <== mon.2 v1:172.21.15.17:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f2c7c001070 con 0x7f2c840e2530 2024-03-29T11:21:33.797 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.815+0000 7f2c81ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:33.797 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.815+0000 7f2c81ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:33.797 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.815+0000 7f2c81ffb640 10 monclient: dump: 2024-03-29T11:21:33.797 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:21:33.797 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:21:33.797 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:33.797 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:33.797 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:33.797 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:33.797 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:21:33.797 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:21:33.797 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:21:33.797 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:33.798 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.815+0000 7f2c921e3640 10 client.4337.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:33.798 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.815+0000 7f2c921e3640 10 monclient: _renew_subs 2024-03-29T11:21:33.798 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.815+0000 7f2c921e3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:21:33.798 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.815+0000 7f2c921e3640 1 -- 172.21.15.17:0/487192413 --> v1:172.21.15.17:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c840e1a50 con 0x7f2c840e2530 2024-03-29T11:21:33.798 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.816+0000 7f2c921e3640 1 librados: init done 2024-03-29T11:21:33.798 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.816+0000 7f2c837fe640 4 set_mon_vals no callback set 2024-03-29T11:21:33.799 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.818+0000 7f2c921e3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:33.799 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.818+0000 7f2c921e3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:33.799 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.818+0000 7f2c921e3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:21:33.800 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.818+0000 7f2c921e3640 1 -- 172.21.15.17:0/487192413 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c840de130 con 0x7f2c840e2530 2024-03-29T11:21:33.800 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.820+0000 7f2c81ffb640 1 -- 172.21.15.17:0/487192413 <== mon.2 v1:172.21.15.17:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f2c7c0011d0 con 0x7f2c840e2530 2024-03-29T11:21:33.800 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.821+0000 7f2c81ffb640 1 -- 172.21.15.17:0/487192413 <== mon.2 v1:172.21.15.17:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f2c7c0848b0 con 0x7f2c840e2530 2024-03-29T11:21:33.800 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.821+0000 7f2c81ffb640 10 client.4337.objecter ms_dispatch 0x7f2c8c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:33.800 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.821+0000 7f2c81ffb640 3 client.4337.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:33.800 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.821+0000 7f2c81ffb640 3 client.4337.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:33.800 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.821+0000 7f2c81ffb640 20 client.4337.objecter dump_active .. 0 homeless 2024-03-29T11:21:33.800 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.822+0000 7f2c911e1640 10 monclient: get_auth_request con 0x7f2c7006dee0 auth_method 0 2024-03-29T11:21:33.800 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.823+0000 7f2c81ffb640 10 client.4337.objecter ms_handle_connect 0x7f2c7006dee0 2024-03-29T11:21:33.804 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.828+0000 7f2c81ffb640 1 -- 172.21.15.17:0/487192413 <== mon.2 v1:172.21.15.17:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2c7c050190 con 0x7f2c840e2530 2024-03-29T11:21:33.804 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.828+0000 7f2c81ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:33.804 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:33.828+0000 7f2c81ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:34.136 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:34.159+0000 7f2c921e3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:34.136 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:34.159+0000 7f2c921e3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:34.136 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:34.159+0000 7f2c921e3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:21:34.136 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:34.159+0000 7f2c921e3640 1 -- 172.21.15.17:0/487192413 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2c840de450 con 0x7f2c840e2530 2024-03-29T11:21:34.137 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:34.160+0000 7f2c81ffb640 1 -- 172.21.15.17:0/487192413 <== mon.2 v1:172.21.15.17:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f2c7c0520c0 con 0x7f2c840e2530 2024-03-29T11:21:34.137 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:34.160+0000 7f2c81ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:34.137 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:34.160+0000 7f2c81ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:34.138 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:21:34.138 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:34.143 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:34.165+0000 7f2c921e3640 10 librados: watch_flush enter 2024-03-29T11:21:34.143 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:34.165+0000 7f2c921e3640 10 librados: watch_flush exit 2024-03-29T11:21:34.143 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:34.165+0000 7f2c921e3640 20 client.4337.objecter shutdown clearing up homeless session... 2024-03-29T11:21:34.143 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:34.165+0000 7f2c921e3640 10 client.4337.objecter successfully canceled tick 2024-03-29T11:21:34.143 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:34.166+0000 7f2c921e3640 1 -- 172.21.15.17:0/487192413 >> v1:172.21.15.17:6816/3236185052 conn(0x7f2c7006dee0 legacy=0x7f2c700703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:34.144 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:34.166+0000 7f2c921e3640 10 monclient: shutdown 2024-03-29T11:21:34.144 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:34.166+0000 7f2c921e3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:34.144 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:34.166+0000 7f2c921e3640 1 -- 172.21.15.17:0/487192413 >> v1:172.21.15.17:6790/0 conn(0x7f2c840e2530 legacy=0x7f2c84189880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:34.144 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:34.166+0000 7f2c921e3640 1 -- 172.21.15.17:0/487192413 shutdown_connections 2024-03-29T11:21:34.144 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:34.166+0000 7f2c921e3640 1 -- 172.21.15.17:0/487192413 >> 172.21.15.17:0/487192413 conn(0x7f2c840058f0 msgr2=0x7f2c840043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:34.144 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:34.166+0000 7f2c921e3640 1 -- 172.21.15.17:0/487192413 shutdown_connections 2024-03-29T11:21:34.144 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:34.166+0000 7f2c921e3640 1 -- 172.21.15.17:0/487192413 wait complete. 2024-03-29T11:21:34.147 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:34.167+0000 7f2c921e3640 1 librados: shutdown 2024-03-29T11:21:34.159 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:40.966 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:41.124 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.146+0000 7f3792559640 10 monclient: get_monmap_and_config 2024-03-29T11:21:41.124 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.146+0000 7f3792559640 10 monclient: build_initial_monmap 2024-03-29T11:21:41.124 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.146+0000 7f3792559640 10 monclient: monmap: 2024-03-29T11:21:41.124 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:21:41.125 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:21:41.125 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:21:41.147891+0000 2024-03-29T11:21:41.125 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:21:41.147891+0000 2024-03-29T11:21:41.125 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:41.125 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:41.125 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:21:41.125 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:21:41.125 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:21:41.125 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:41.125 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.149+0000 7f3792559640 1 Processor -- start 2024-03-29T11:21:41.127 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.149+0000 7f3792559640 1 -- start start 2024-03-29T11:21:41.127 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.149+0000 7f3792559640 10 monclient: init 2024-03-29T11:21:41.127 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.150+0000 7f3792559640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:41.128 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.150+0000 7f3792559640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:21:41.128 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.150+0000 7f3792559640 10 monclient(hunting): picked mon.noname-a con 0x7f378c0b4100 addr v1:172.21.15.17:6789/0 2024-03-29T11:21:41.128 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.150+0000 7f3792559640 10 monclient(hunting): picked mon.noname-b con 0x7f378c13ee90 addr v1:172.21.15.17:6790/0 2024-03-29T11:21:41.128 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.150+0000 7f3792559640 10 monclient(hunting): picked mon.noname-c con 0x7f378c0af900 addr v1:172.21.15.94:6789/0 2024-03-29T11:21:41.128 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.150+0000 7f3792559640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f378c0afd10 con 0x7f378c0b4100 2024-03-29T11:21:41.128 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.150+0000 7f3792559640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f378c0af6f0 con 0x7f378c0af900 2024-03-29T11:21:41.128 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.150+0000 7f3792559640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f378c0aac80 con 0x7f378c13ee90 2024-03-29T11:21:41.128 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.150+0000 7f3792559640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:41.128 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.150+0000 7f3792559640 10 monclient(hunting): authenticate will time out at 1007.355664s 2024-03-29T11:21:41.132 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.155+0000 7f3790af6640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f378c0af900 0x7f378c14ac10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:38361/0 (socket says 172.21.15.17:38361) 2024-03-29T11:21:41.132 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.156+0000 7f3790af6640 1 -- 172.21.15.17:0/3774911638 learned_addr learned my addr 172.21.15.17:0/3774911638 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:21:41.136 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.156+0000 7f378affd640 1 -- 172.21.15.17:0/3774911638 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2628420075 0 0) 0x7f378c0aac80 con 0x7f378c13ee90 2024-03-29T11:21:41.136 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.156+0000 7f378affd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:41.137 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.156+0000 7f378affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:41.137 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.156+0000 7f378affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:41.137 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.156+0000 7f378affd640 10 monclient(hunting): my global_id is 4340 2024-03-29T11:21:41.137 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.156+0000 7f378affd640 1 -- 172.21.15.17:0/3774911638 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3774003520 con 0x7f378c13ee90 2024-03-29T11:21:41.137 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.156+0000 7f378affd640 1 -- 172.21.15.17:0/3774911638 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1145414570 0 0) 0x7f378c0af6f0 con 0x7f378c0af900 2024-03-29T11:21:41.137 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.156+0000 7f378affd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:41.137 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.156+0000 7f378affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:41.137 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.156+0000 7f378affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:41.137 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.156+0000 7f378affd640 10 monclient(hunting): my global_id is 4324 2024-03-29T11:21:41.137 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.157+0000 7f378affd640 1 -- 172.21.15.17:0/3774911638 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f378c0aac80 con 0x7f378c0af900 2024-03-29T11:21:41.137 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.157+0000 7f378affd640 1 -- 172.21.15.17:0/3774911638 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1023086498 0 0) 0x7f3774003520 con 0x7f378c13ee90 2024-03-29T11:21:41.137 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.157+0000 7f378affd640 1 -- 172.21.15.17:0/3774911638 >> v1:172.21.15.94:6789/0 conn(0x7f378c0af900 legacy=0x7f378c14ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:41.137 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.157+0000 7f378affd640 1 -- 172.21.15.17:0/3774911638 >> v1:172.21.15.17:6789/0 conn(0x7f378c0b4100 legacy=0x7f378c0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:41.137 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.157+0000 7f378affd640 10 monclient: _finish_hunting 0 2024-03-29T11:21:41.138 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.157+0000 7f378affd640 1 monclient: found mon.noname-b 2024-03-29T11:21:41.138 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.157+0000 7f378affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:41.138 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.157+0000 7f378affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.17:6790/0 2024-03-29T11:21:41.138 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.157+0000 7f378affd640 1 -- 172.21.15.17:0/3774911638 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f378c0aaf60 con 0x7f378c13ee90 2024-03-29T11:21:41.138 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.157+0000 7f378affd640 10 monclient: _finish_auth 0 2024-03-29T11:21:41.138 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.157+0000 7f378affd640 10 monclient: _check_auth_tickets 2024-03-29T11:21:41.138 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.157+0000 7f378affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:41.138 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.158+0000 7f3792559640 5 monclient: authenticate success, global_id 4340 2024-03-29T11:21:41.138 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.158+0000 7f3792559640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:41.138 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.158+0000 7f378affd640 1 -- 172.21.15.17:0/3774911638 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f3784001a10 con 0x7f378c13ee90 2024-03-29T11:21:41.138 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.158+0000 7f378affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:41.138 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.158+0000 7f378affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:41.138 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.158+0000 7f378affd640 10 monclient: dump: 2024-03-29T11:21:41.138 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:21:41.139 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:21:41.139 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:41.139 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:41.139 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:41.139 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:41.139 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:21:41.139 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:21:41.139 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:21:41.139 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:41.139 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.158+0000 7f3792559640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:41.139 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.158+0000 7f378affd640 1 -- 172.21.15.17:0/3774911638 <== mon.2 v1:172.21.15.17:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3784001c30 con 0x7f378c13ee90 2024-03-29T11:21:41.139 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.158+0000 7f378affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:41.139 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.158+0000 7f3792559640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:41.139 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.158+0000 7f3792559640 4 set_mon_vals no callback set 2024-03-29T11:21:41.140 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.158+0000 7f378affd640 1 -- 172.21.15.17:0/3774911638 <== mon.2 v1:172.21.15.17:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f3784001fc0 con 0x7f378c13ee90 2024-03-29T11:21:41.140 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.158+0000 7f378affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:41.140 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.158+0000 7f378affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:41.140 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.158+0000 7f378affd640 10 monclient: dump: 2024-03-29T11:21:41.140 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:21:41.140 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:21:41.140 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:41.140 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:41.140 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:41.140 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:41.140 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:21:41.140 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:21:41.140 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:21:41.141 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:41.141 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.159+0000 7f3792559640 10 monclient: shutdown 2024-03-29T11:21:41.141 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.159+0000 7f3792559640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:41.141 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.159+0000 7f3792559640 1 -- 172.21.15.17:0/3774911638 >> v1:172.21.15.17:6790/0 conn(0x7f378c13ee90 legacy=0x7f378c13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:41.141 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.159+0000 7f3792559640 1 -- 172.21.15.17:0/3774911638 shutdown_connections 2024-03-29T11:21:41.141 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.159+0000 7f3792559640 1 -- 172.21.15.17:0/3774911638 >> 172.21.15.17:0/3774911638 conn(0x7f378c0b35e0 msgr2=0x7f378c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:41.141 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.159+0000 7f3792559640 1 -- 172.21.15.17:0/3774911638 shutdown_connections 2024-03-29T11:21:41.141 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.159+0000 7f3792559640 1 -- 172.21.15.17:0/3774911638 wait complete. 2024-03-29T11:21:41.141 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.160+0000 7f3792559640 10 monclient: build_initial_monmap 2024-03-29T11:21:41.141 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.160+0000 7f3792559640 10 monclient: monmap: 2024-03-29T11:21:41.142 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:21:41.142 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:41.142 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:21:41.142 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:21:41.142 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:41.142 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:41.142 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:21:41.142 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:21:41.142 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:21:41.142 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:41.142 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.161+0000 7f3792559640 1 librados: starting msgr at 2024-03-29T11:21:41.142 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.161+0000 7f3792559640 1 librados: starting objecter 2024-03-29T11:21:41.142 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.161+0000 7f3792559640 1 Processor -- start 2024-03-29T11:21:41.142 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.161+0000 7f3792559640 1 -- start start 2024-03-29T11:21:41.143 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.161+0000 7f3792559640 1 librados: setting wanted keys 2024-03-29T11:21:41.143 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.161+0000 7f3792559640 1 librados: calling monclient init 2024-03-29T11:21:41.143 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.161+0000 7f3792559640 10 monclient: init 2024-03-29T11:21:41.143 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.162+0000 7f3792559640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:41.143 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.162+0000 7f3792559640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:21:41.143 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.162+0000 7f3792559640 10 monclient(hunting): picked mon.noname-a con 0x7f378c0af900 addr v1:172.21.15.17:6789/0 2024-03-29T11:21:41.143 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.162+0000 7f3792559640 10 monclient(hunting): picked mon.noname-b con 0x7f378c0b4100 addr v1:172.21.15.94:6789/0 2024-03-29T11:21:41.143 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.162+0000 7f3792559640 10 monclient(hunting): picked mon.noname-c con 0x7f378c13ee90 addr v1:172.21.15.17:6790/0 2024-03-29T11:21:41.143 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.162+0000 7f3792559640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f378c070d50 con 0x7f378c0af900 2024-03-29T11:21:41.143 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.162+0000 7f3792559640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f378c12cf10 con 0x7f378c0b4100 2024-03-29T11:21:41.143 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.162+0000 7f3792559640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f378c072200 con 0x7f378c13ee90 2024-03-29T11:21:41.143 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.162+0000 7f3792559640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:41.143 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.162+0000 7f3792559640 10 monclient(hunting): authenticate will time out at 1007.367204s 2024-03-29T11:21:41.144 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.166+0000 7f378b7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f378c0b4100 0x7f378c12c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:38459/0 (socket says 172.21.15.17:38459) 2024-03-29T11:21:41.144 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.166+0000 7f378b7fe640 1 -- 172.21.15.17:0/380345320 learned_addr learned my addr 172.21.15.17:0/380345320 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:21:41.144 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.167+0000 7f3788ff9640 10 client.?.objecter ms_handle_connect 0x7f378c0b4100 2024-03-29T11:21:41.144 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.167+0000 7f3788ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:41.145 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.168+0000 7f3788ff9640 10 client.?.objecter ms_handle_connect 0x7f378c13ee90 2024-03-29T11:21:41.145 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.168+0000 7f3788ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:41.150 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.169+0000 7f3788ff9640 10 client.?.objecter ms_handle_connect 0x7f378c0af900 2024-03-29T11:21:41.150 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.169+0000 7f3788ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:41.150 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.169+0000 7f3788ff9640 1 -- 172.21.15.17:0/380345320 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2178027020 0 0) 0x7f378c072200 con 0x7f378c13ee90 2024-03-29T11:21:41.150 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.169+0000 7f3788ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:21:41.150 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.169+0000 7f3788ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:41.150 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.169+0000 7f3788ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:41.150 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.169+0000 7f3788ff9640 10 monclient(hunting): my global_id is 4343 2024-03-29T11:21:41.151 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.169+0000 7f3788ff9640 1 -- 172.21.15.17:0/380345320 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3778004eb0 con 0x7f378c13ee90 2024-03-29T11:21:41.151 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.169+0000 7f3788ff9640 1 -- 172.21.15.17:0/380345320 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3540114359 0 0) 0x7f378c12cf10 con 0x7f378c0b4100 2024-03-29T11:21:41.151 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.169+0000 7f3788ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:21:41.151 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.169+0000 7f3788ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:41.151 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.169+0000 7f3788ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:41.151 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.169+0000 7f3788ff9640 10 monclient(hunting): my global_id is 4327 2024-03-29T11:21:41.151 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.169+0000 7f3788ff9640 1 -- 172.21.15.17:0/380345320 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f378c072200 con 0x7f378c0b4100 2024-03-29T11:21:41.151 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.170+0000 7f3788ff9640 1 -- 172.21.15.17:0/380345320 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2614250910 0 0) 0x7f378c070d50 con 0x7f378c0af900 2024-03-29T11:21:41.151 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.170+0000 7f3788ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:21:41.151 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.170+0000 7f3788ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:41.151 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.170+0000 7f3788ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:41.151 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.170+0000 7f3788ff9640 10 monclient(hunting): my global_id is 4329 2024-03-29T11:21:41.151 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.170+0000 7f3788ff9640 1 -- 172.21.15.17:0/380345320 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f378c12cf10 con 0x7f378c0af900 2024-03-29T11:21:41.151 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.170+0000 7f3788ff9640 1 -- 172.21.15.17:0/380345320 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3639862 0 0) 0x7f378c072200 con 0x7f378c0b4100 2024-03-29T11:21:41.152 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.170+0000 7f3788ff9640 1 -- 172.21.15.17:0/380345320 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f378c070d50 con 0x7f378c0b4100 2024-03-29T11:21:41.152 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.170+0000 7f3788ff9640 1 -- 172.21.15.17:0/380345320 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f3784002000 con 0x7f378c0b4100 2024-03-29T11:21:41.152 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.171+0000 7f3788ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:41.152 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.171+0000 7f3788ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:41.152 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.171+0000 7f3788ff9640 10 monclient(hunting): dump: 2024-03-29T11:21:41.152 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:21:41.152 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:21:41.152 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:41.152 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:41.152 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:41.152 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:41.152 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:21:41.152 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:21:41.152 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:21:41.153 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:41.153 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.171+0000 7f3788ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:41.153 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.171+0000 7f3788ff9640 1 -- 172.21.15.17:0/380345320 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3521011849 0 0) 0x7f378c12cf10 con 0x7f378c0af900 2024-03-29T11:21:41.153 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.171+0000 7f3788ff9640 1 -- 172.21.15.17:0/380345320 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f378c072200 con 0x7f378c0af900 2024-03-29T11:21:41.153 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.171+0000 7f3788ff9640 1 -- 172.21.15.17:0/380345320 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f3780001490 con 0x7f378c0af900 2024-03-29T11:21:41.153 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.171+0000 7f3788ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:41.153 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.171+0000 7f3788ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:41.153 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.171+0000 7f3788ff9640 10 monclient(hunting): dump: 2024-03-29T11:21:41.153 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:21:41.153 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:21:41.153 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:41.153 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:41.153 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:41.153 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:41.154 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:21:41.154 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:21:41.154 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:21:41.154 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:41.154 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.171+0000 7f3788ff9640 1 -- 172.21.15.17:0/380345320 <== mon.1 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2012267040 0 0) 0x7f378c070d50 con 0x7f378c0b4100 2024-03-29T11:21:41.154 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.171+0000 7f3788ff9640 1 -- 172.21.15.17:0/380345320 >> v1:172.21.15.17:6790/0 conn(0x7f378c13ee90 legacy=0x7f378c1ef5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:41.154 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.171+0000 7f3788ff9640 1 -- 172.21.15.17:0/380345320 >> v1:172.21.15.17:6789/0 conn(0x7f378c0af900 legacy=0x7f378c0714c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:41.154 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.171+0000 7f3788ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:21:41.154 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.171+0000 7f3788ff9640 1 monclient: found mon.b 2024-03-29T11:21:41.154 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.171+0000 7f3788ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:41.154 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.172+0000 7f3788ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:21:41.154 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.172+0000 7f3788ff9640 1 -- 172.21.15.17:0/380345320 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f378c0724e0 con 0x7f378c0b4100 2024-03-29T11:21:41.154 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.172+0000 7f3788ff9640 10 monclient: _finish_auth 0 2024-03-29T11:21:41.154 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.172+0000 7f3788ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:21:41.155 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.172+0000 7f3788ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:41.155 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.172+0000 7f3792559640 5 monclient: authenticate success, global_id 4327 2024-03-29T11:21:41.155 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.172+0000 7f3792559640 10 monclient: _renew_subs 2024-03-29T11:21:41.155 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.172+0000 7f3792559640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:21:41.155 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.172+0000 7f3792559640 1 -- 172.21.15.17:0/380345320 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f378c12c3e0 con 0x7f378c0b4100 2024-03-29T11:21:41.155 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.172+0000 7f3792559640 10 client.4327.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:41.155 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.172+0000 7f3792559640 10 monclient: _renew_subs 2024-03-29T11:21:41.155 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.172+0000 7f3792559640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:21:41.155 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.172+0000 7f3792559640 1 -- 172.21.15.17:0/380345320 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f378c141c90 con 0x7f378c0b4100 2024-03-29T11:21:41.155 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.172+0000 7f3792559640 1 librados: init done 2024-03-29T11:21:41.157 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.173+0000 7f3788ff9640 1 -- 172.21.15.17:0/380345320 <== mon.1 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3784003c20 con 0x7f378c0b4100 2024-03-29T11:21:41.157 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.173+0000 7f3788ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:41.157 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.173+0000 7f3788ff9640 1 -- 172.21.15.17:0/380345320 <== mon.1 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f3784003ff0 con 0x7f378c0b4100 2024-03-29T11:21:41.157 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.173+0000 7f3788ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:41.157 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.173+0000 7f3788ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:41.157 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.173+0000 7f3788ff9640 10 monclient: dump: 2024-03-29T11:21:41.157 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:21:41.158 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:21:41.158 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:41.158 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:41.158 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:41.158 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:41.158 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:21:41.158 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:21:41.158 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:21:41.158 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:41.158 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.173+0000 7f378a7fc640 4 set_mon_vals no callback set 2024-03-29T11:21:41.158 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.175+0000 7f3792559640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:41.158 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.175+0000 7f3792559640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:41.158 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.175+0000 7f3792559640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:21:41.158 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.175+0000 7f3792559640 1 -- 172.21.15.17:0/380345320 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f376c005080 con 0x7f378c0b4100 2024-03-29T11:21:41.159 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.178+0000 7f3788ff9640 1 -- 172.21.15.17:0/380345320 <== mon.1 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f3784004150 con 0x7f378c0b4100 2024-03-29T11:21:41.159 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.178+0000 7f3788ff9640 1 -- 172.21.15.17:0/380345320 <== mon.1 v1:172.21.15.94:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f3784084e60 con 0x7f378c0b4100 2024-03-29T11:21:41.159 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.178+0000 7f3788ff9640 10 client.4327.objecter ms_dispatch 0x7f378c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:41.159 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.178+0000 7f3788ff9640 3 client.4327.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:41.159 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.178+0000 7f3788ff9640 3 client.4327.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:41.159 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.178+0000 7f3788ff9640 20 client.4327.objecter dump_active .. 0 homeless 2024-03-29T11:21:41.159 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.180+0000 7f378bfff640 10 monclient: get_auth_request con 0x7f377806e010 auth_method 0 2024-03-29T11:21:41.159 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.180+0000 7f3788ff9640 10 client.4327.objecter ms_handle_connect 0x7f377806e010 2024-03-29T11:21:41.162 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.186+0000 7f3788ff9640 1 -- 172.21.15.17:0/380345320 <== mon.1 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3784050740 con 0x7f378c0b4100 2024-03-29T11:21:41.162 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.186+0000 7f3788ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:41.162 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.186+0000 7f3788ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:41.488 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.512+0000 7f3792559640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:41.488 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.512+0000 7f3792559640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:41.488 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.512+0000 7f3792559640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:21:41.488 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.512+0000 7f3792559640 1 -- 172.21.15.17:0/380345320 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f376c005330 con 0x7f378c0b4100 2024-03-29T11:21:41.489 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.513+0000 7f3788ff9640 1 -- 172.21.15.17:0/380345320 <== mon.1 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f3784052670 con 0x7f378c0b4100 2024-03-29T11:21:41.489 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.513+0000 7f3788ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:41.489 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.513+0000 7f3788ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:41.492 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:21:41.492 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:41.493 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.516+0000 7f3792559640 10 librados: watch_flush enter 2024-03-29T11:21:41.493 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.516+0000 7f3792559640 10 librados: watch_flush exit 2024-03-29T11:21:41.493 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.516+0000 7f3792559640 20 client.4327.objecter shutdown clearing up homeless session... 2024-03-29T11:21:41.493 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.516+0000 7f3792559640 10 client.4327.objecter successfully canceled tick 2024-03-29T11:21:41.493 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.517+0000 7f3792559640 1 -- 172.21.15.17:0/380345320 >> v1:172.21.15.17:6816/3236185052 conn(0x7f377806e010 legacy=0x7f37780704d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:41.493 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.517+0000 7f3792559640 10 monclient: shutdown 2024-03-29T11:21:41.493 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.517+0000 7f3792559640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:41.493 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.517+0000 7f3792559640 1 -- 172.21.15.17:0/380345320 >> v1:172.21.15.94:6789/0 conn(0x7f378c0b4100 legacy=0x7f378c12c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:41.494 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.517+0000 7f3792559640 1 -- 172.21.15.17:0/380345320 shutdown_connections 2024-03-29T11:21:41.494 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.517+0000 7f3792559640 1 -- 172.21.15.17:0/380345320 >> 172.21.15.17:0/380345320 conn(0x7f378c0b35e0 msgr2=0x7f378c0ad0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:41.494 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.517+0000 7f3792559640 1 -- 172.21.15.17:0/380345320 shutdown_connections 2024-03-29T11:21:41.494 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.518+0000 7f3792559640 1 -- 172.21.15.17:0/380345320 wait complete. 2024-03-29T11:21:41.495 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:41.518+0000 7f3792559640 1 librados: shutdown 2024-03-29T11:21:41.513 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:48.320 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:48.466 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.489+0000 7eff9cbf4640 10 monclient: get_monmap_and_config 2024-03-29T11:21:48.466 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.489+0000 7eff9cbf4640 10 monclient: build_initial_monmap 2024-03-29T11:21:48.466 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.489+0000 7eff9cbf4640 10 monclient: monmap: 2024-03-29T11:21:48.467 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:21:48.467 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:21:48.467 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:21:48.490253+0000 2024-03-29T11:21:48.467 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:21:48.490253+0000 2024-03-29T11:21:48.467 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:48.467 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:48.467 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:21:48.468 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:21:48.468 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:21:48.468 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:48.468 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.491+0000 7eff9cbf4640 1 Processor -- start 2024-03-29T11:21:48.470 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.494+0000 7eff9cbf4640 1 -- start start 2024-03-29T11:21:48.470 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.494+0000 7eff9cbf4640 10 monclient: init 2024-03-29T11:21:48.471 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.495+0000 7eff9cbf4640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:48.471 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.495+0000 7eff9cbf4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:21:48.473 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.495+0000 7eff9cbf4640 10 monclient(hunting): picked mon.noname-c con 0x7eff980b4100 addr v1:172.21.15.94:6789/0 2024-03-29T11:21:48.473 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.496+0000 7eff9cbf4640 10 monclient(hunting): picked mon.noname-b con 0x7eff9813ee90 addr v1:172.21.15.17:6790/0 2024-03-29T11:21:48.473 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.496+0000 7eff9cbf4640 10 monclient(hunting): picked mon.noname-a con 0x7eff980af900 addr v1:172.21.15.17:6789/0 2024-03-29T11:21:48.473 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.496+0000 7eff9cbf4640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff980afd10 con 0x7eff980af900 2024-03-29T11:21:48.474 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.496+0000 7eff9cbf4640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff980af6f0 con 0x7eff980b4100 2024-03-29T11:21:48.474 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.496+0000 7eff9cbf4640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff980aaf60 con 0x7eff9813ee90 2024-03-29T11:21:48.474 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.496+0000 7eff9cbf4640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:48.474 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.496+0000 7eff9cbf4640 10 monclient(hunting): authenticate will time out at 1014.701048s 2024-03-29T11:21:48.475 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.499+0000 7eff9659c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7eff980b4100 0x7eff980a9850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:33139/0 (socket says 172.21.15.17:33139) 2024-03-29T11:21:48.476 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.499+0000 7eff9659c640 1 -- 172.21.15.17:0/521066312 learned_addr learned my addr 172.21.15.17:0/521066312 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:21:48.478 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.501+0000 7eff9559a640 1 -- 172.21.15.17:0/521066312 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2611619669 0 0) 0x7eff980aaf60 con 0x7eff9813ee90 2024-03-29T11:21:48.478 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.501+0000 7eff9559a640 10 monclient(hunting): _negotiate 2024-03-29T11:21:48.479 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.501+0000 7eff9559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:48.479 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.501+0000 7eff9559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:48.479 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.501+0000 7eff9559a640 10 monclient(hunting): my global_id is 4346 2024-03-29T11:21:48.479 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.501+0000 7eff9559a640 1 -- 172.21.15.17:0/521066312 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff80003520 con 0x7eff9813ee90 2024-03-29T11:21:48.479 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.501+0000 7eff9559a640 1 -- 172.21.15.17:0/521066312 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1738316349 0 0) 0x7eff980af6f0 con 0x7eff980b4100 2024-03-29T11:21:48.479 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.501+0000 7eff9559a640 10 monclient(hunting): _negotiate 2024-03-29T11:21:48.479 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.501+0000 7eff9559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:48.479 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.501+0000 7eff9559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:48.479 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.501+0000 7eff9559a640 10 monclient(hunting): my global_id is 4330 2024-03-29T11:21:48.479 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.501+0000 7eff9559a640 1 -- 172.21.15.17:0/521066312 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff980aaf60 con 0x7eff980b4100 2024-03-29T11:21:48.479 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.502+0000 7eff9559a640 1 -- 172.21.15.17:0/521066312 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1082227546 0 0) 0x7eff980aaf60 con 0x7eff980b4100 2024-03-29T11:21:48.479 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.502+0000 7eff9559a640 1 -- 172.21.15.17:0/521066312 >> v1:172.21.15.17:6790/0 conn(0x7eff9813ee90 legacy=0x7eff9813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:48.480 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.502+0000 7eff9559a640 1 -- 172.21.15.17:0/521066312 >> v1:172.21.15.17:6789/0 conn(0x7eff980af900 legacy=0x7eff98149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:48.480 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.502+0000 7eff9559a640 10 monclient: _finish_hunting 0 2024-03-29T11:21:48.480 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.502+0000 7eff9559a640 1 monclient: found mon.noname-c 2024-03-29T11:21:48.480 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.502+0000 7eff9559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:48.480 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.502+0000 7eff9559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6789/0 2024-03-29T11:21:48.480 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.502+0000 7eff9559a640 1 -- 172.21.15.17:0/521066312 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff9813e970 con 0x7eff980b4100 2024-03-29T11:21:48.480 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.502+0000 7eff9559a640 10 monclient: _finish_auth 0 2024-03-29T11:21:48.480 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.502+0000 7eff9559a640 10 monclient: _check_auth_tickets 2024-03-29T11:21:48.480 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.502+0000 7eff9559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:48.480 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.502+0000 7eff9559a640 1 -- 172.21.15.17:0/521066312 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7eff8c0015f0 con 0x7eff980b4100 2024-03-29T11:21:48.480 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.502+0000 7eff9559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:48.480 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.502+0000 7eff9559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:48.480 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.502+0000 7eff9559a640 10 monclient: dump: 2024-03-29T11:21:48.480 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:21:48.480 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:21:48.481 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:48.481 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:48.481 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:48.481 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:48.481 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:21:48.481 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:21:48.481 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:21:48.481 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:48.482 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.502+0000 7eff9559a640 1 -- 172.21.15.17:0/521066312 <== mon.1 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7eff8c001750 con 0x7eff980b4100 2024-03-29T11:21:48.483 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.502+0000 7eff9559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:48.483 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.502+0000 7eff9559a640 1 -- 172.21.15.17:0/521066312 <== mon.1 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7eff8c001b60 con 0x7eff980b4100 2024-03-29T11:21:48.483 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.502+0000 7eff9559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:48.483 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.502+0000 7eff9559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:48.483 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.502+0000 7eff9559a640 10 monclient: dump: 2024-03-29T11:21:48.483 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:21:48.483 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:21:48.483 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:48.483 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:48.483 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:48.483 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:48.484 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:21:48.484 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:21:48.484 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:21:48.484 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:48.484 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.503+0000 7eff9cbf4640 5 monclient: authenticate success, global_id 4330 2024-03-29T11:21:48.484 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.503+0000 7eff9cbf4640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:48.484 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.503+0000 7eff9cbf4640 4 set_mon_vals no callback set 2024-03-29T11:21:48.484 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.504+0000 7eff9cbf4640 10 monclient: shutdown 2024-03-29T11:21:48.484 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.504+0000 7eff9cbf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:48.484 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.504+0000 7eff9cbf4640 1 -- 172.21.15.17:0/521066312 >> v1:172.21.15.94:6789/0 conn(0x7eff980b4100 legacy=0x7eff980a9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:48.484 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.504+0000 7eff9cbf4640 1 -- 172.21.15.17:0/521066312 shutdown_connections 2024-03-29T11:21:48.484 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.504+0000 7eff9cbf4640 1 -- 172.21.15.17:0/521066312 >> 172.21.15.17:0/521066312 conn(0x7eff980b35e0 msgr2=0x7eff980b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:48.484 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.505+0000 7eff9cbf4640 1 -- 172.21.15.17:0/521066312 shutdown_connections 2024-03-29T11:21:48.485 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.505+0000 7eff9cbf4640 1 -- 172.21.15.17:0/521066312 wait complete. 2024-03-29T11:21:48.485 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.506+0000 7eff9cbf4640 10 monclient: build_initial_monmap 2024-03-29T11:21:48.485 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.506+0000 7eff9cbf4640 10 monclient: monmap: 2024-03-29T11:21:48.485 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:21:48.485 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:48.485 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:21:48.485 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:21:48.485 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:48.485 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:48.485 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:21:48.485 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:21:48.486 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:21:48.486 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:48.486 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.507+0000 7eff9cbf4640 1 librados: starting msgr at 2024-03-29T11:21:48.486 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.507+0000 7eff9cbf4640 1 librados: starting objecter 2024-03-29T11:21:48.486 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.507+0000 7eff9cbf4640 1 Processor -- start 2024-03-29T11:21:48.486 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.507+0000 7eff9cbf4640 1 -- start start 2024-03-29T11:21:48.486 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.507+0000 7eff9cbf4640 1 librados: setting wanted keys 2024-03-29T11:21:48.486 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.507+0000 7eff9cbf4640 1 librados: calling monclient init 2024-03-29T11:21:48.486 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.507+0000 7eff9cbf4640 10 monclient: init 2024-03-29T11:21:48.487 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.510+0000 7eff9cbf4640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:48.488 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.510+0000 7eff9cbf4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:21:48.488 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.510+0000 7eff9cbf4640 10 monclient(hunting): picked mon.noname-c con 0x7eff980af900 addr v1:172.21.15.17:6790/0 2024-03-29T11:21:48.488 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.510+0000 7eff9cbf4640 10 monclient(hunting): picked mon.noname-a con 0x7eff980b4100 addr v1:172.21.15.17:6789/0 2024-03-29T11:21:48.488 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.510+0000 7eff9cbf4640 10 monclient(hunting): picked mon.noname-b con 0x7eff9813ee90 addr v1:172.21.15.94:6789/0 2024-03-29T11:21:48.488 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.510+0000 7eff9cbf4640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff98150a50 con 0x7eff980b4100 2024-03-29T11:21:48.488 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.510+0000 7eff9cbf4640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff98150f00 con 0x7eff9813ee90 2024-03-29T11:21:48.488 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.510+0000 7eff9cbf4640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff98128db0 con 0x7eff980af900 2024-03-29T11:21:48.488 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.510+0000 7eff9cbf4640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:48.488 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.510+0000 7eff9cbf4640 10 monclient(hunting): authenticate will time out at 1014.715218s 2024-03-29T11:21:48.491 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.514+0000 7eff9659c640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7eff980af900 0x7eff981501c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:59295/0 (socket says 172.21.15.17:59295) 2024-03-29T11:21:48.491 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.514+0000 7eff9659c640 1 -- 172.21.15.17:0/2212135310 learned_addr learned my addr 172.21.15.17:0/2212135310 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:21:48.497 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.519+0000 7eff86ffd640 10 client.?.objecter ms_handle_connect 0x7eff980af900 2024-03-29T11:21:48.497 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.519+0000 7eff86ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:48.497 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.519+0000 7eff86ffd640 10 client.?.objecter ms_handle_connect 0x7eff980b4100 2024-03-29T11:21:48.497 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.519+0000 7eff86ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:48.497 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.519+0000 7eff86ffd640 1 -- 172.21.15.17:0/2212135310 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3557712987 0 0) 0x7eff98128db0 con 0x7eff980af900 2024-03-29T11:21:48.497 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.519+0000 7eff86ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:48.498 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.520+0000 7eff86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:48.498 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.520+0000 7eff86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:48.498 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.520+0000 7eff86ffd640 10 monclient(hunting): my global_id is 4349 2024-03-29T11:21:48.498 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.520+0000 7eff86ffd640 1 -- 172.21.15.17:0/2212135310 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff78004eb0 con 0x7eff980af900 2024-03-29T11:21:48.498 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.520+0000 7eff86ffd640 1 -- 172.21.15.17:0/2212135310 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3538904673 0 0) 0x7eff98150a50 con 0x7eff980b4100 2024-03-29T11:21:48.498 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.520+0000 7eff86ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:48.498 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.521+0000 7eff86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:48.499 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.522+0000 7eff86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:48.499 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.522+0000 7eff86ffd640 10 monclient(hunting): my global_id is 4335 2024-03-29T11:21:48.499 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.522+0000 7eff86ffd640 1 -- 172.21.15.17:0/2212135310 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff98128db0 con 0x7eff980b4100 2024-03-29T11:21:48.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.522+0000 7eff86ffd640 10 client.?.objecter ms_handle_connect 0x7eff9813ee90 2024-03-29T11:21:48.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.522+0000 7eff86ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:48.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.522+0000 7eff86ffd640 1 -- 172.21.15.17:0/2212135310 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 443953858 0 0) 0x7eff78004eb0 con 0x7eff980af900 2024-03-29T11:21:48.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.522+0000 7eff86ffd640 1 -- 172.21.15.17:0/2212135310 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7eff98150a50 con 0x7eff980af900 2024-03-29T11:21:48.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.522+0000 7eff86ffd640 1 -- 172.21.15.17:0/2212135310 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4186517365 0 0) 0x7eff98150f00 con 0x7eff9813ee90 2024-03-29T11:21:48.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.522+0000 7eff86ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:48.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.522+0000 7eff86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:48.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.522+0000 7eff86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:48.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.522+0000 7eff86ffd640 10 monclient(hunting): my global_id is 4333 2024-03-29T11:21:48.501 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.522+0000 7eff86ffd640 1 -- 172.21.15.17:0/2212135310 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff78004eb0 con 0x7eff9813ee90 2024-03-29T11:21:48.501 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.522+0000 7eff86ffd640 1 -- 172.21.15.17:0/2212135310 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7eff8c001b60 con 0x7eff980af900 2024-03-29T11:21:48.501 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.522+0000 7eff86ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:48.501 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.522+0000 7eff86ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:48.501 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.522+0000 7eff86ffd640 10 monclient(hunting): dump: 2024-03-29T11:21:48.501 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:21:48.501 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:21:48.501 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:48.501 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:48.501 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:48.502 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:48.502 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:21:48.502 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:21:48.502 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:21:48.502 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:48.502 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.522+0000 7eff86ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:48.502 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.522+0000 7eff86ffd640 1 -- 172.21.15.17:0/2212135310 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 777960568 0 0) 0x7eff78004eb0 con 0x7eff9813ee90 2024-03-29T11:21:48.502 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.522+0000 7eff86ffd640 1 -- 172.21.15.17:0/2212135310 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7eff98150f00 con 0x7eff9813ee90 2024-03-29T11:21:48.503 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.522+0000 7eff86ffd640 1 -- 172.21.15.17:0/2212135310 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7eff880016a0 con 0x7eff9813ee90 2024-03-29T11:21:48.503 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.522+0000 7eff86ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:48.503 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.522+0000 7eff86ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:48.503 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.522+0000 7eff86ffd640 10 monclient(hunting): dump: 2024-03-29T11:21:48.503 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:21:48.503 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:21:48.503 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:48.503 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:48.504 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:48.504 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:48.504 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:21:48.504 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:21:48.504 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:21:48.504 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:48.504 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.522+0000 7eff86ffd640 1 -- 172.21.15.17:0/2212135310 <== mon.2 v1:172.21.15.17:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4259281848 0 0) 0x7eff98150a50 con 0x7eff980af900 2024-03-29T11:21:48.504 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.522+0000 7eff86ffd640 1 -- 172.21.15.17:0/2212135310 >> v1:172.21.15.94:6789/0 conn(0x7eff9813ee90 legacy=0x7eff981e2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:48.504 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.522+0000 7eff86ffd640 1 -- 172.21.15.17:0/2212135310 >> v1:172.21.15.17:6789/0 conn(0x7eff980b4100 legacy=0x7eff9813d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:48.504 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.522+0000 7eff86ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:21:48.505 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.522+0000 7eff86ffd640 1 monclient: found mon.c 2024-03-29T11:21:48.505 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.522+0000 7eff86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:48.505 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.522+0000 7eff86ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:21:48.505 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.522+0000 7eff86ffd640 1 -- 172.21.15.17:0/2212135310 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff98129090 con 0x7eff980af900 2024-03-29T11:21:48.505 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.522+0000 7eff86ffd640 10 monclient: _finish_auth 0 2024-03-29T11:21:48.505 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.522+0000 7eff86ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:21:48.505 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.523+0000 7eff86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:48.506 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.523+0000 7eff9cbf4640 5 monclient: authenticate success, global_id 4349 2024-03-29T11:21:48.506 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.523+0000 7eff9cbf4640 10 monclient: _renew_subs 2024-03-29T11:21:48.507 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.523+0000 7eff9cbf4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:21:48.507 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.523+0000 7eff9cbf4640 1 -- 172.21.15.17:0/2212135310 --> v1:172.21.15.17:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff9814f9e0 con 0x7eff980af900 2024-03-29T11:21:48.507 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.523+0000 7eff9cbf4640 10 client.4349.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:48.507 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.523+0000 7eff9cbf4640 10 monclient: _renew_subs 2024-03-29T11:21:48.507 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.523+0000 7eff9cbf4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:21:48.507 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.523+0000 7eff9cbf4640 1 -- 172.21.15.17:0/2212135310 --> v1:172.21.15.17:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7eff9814e750 con 0x7eff980af900 2024-03-29T11:21:48.507 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.523+0000 7eff9cbf4640 1 librados: init done 2024-03-29T11:21:48.507 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.523+0000 7eff86ffd640 1 -- 172.21.15.17:0/2212135310 <== mon.2 v1:172.21.15.17:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7eff8c004b00 con 0x7eff980af900 2024-03-29T11:21:48.507 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.523+0000 7eff86ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:48.508 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.523+0000 7eff86ffd640 1 -- 172.21.15.17:0/2212135310 <== mon.2 v1:172.21.15.17:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7eff8c004e10 con 0x7eff980af900 2024-03-29T11:21:48.508 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.523+0000 7eff86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:48.508 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.523+0000 7eff86ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:48.508 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.523+0000 7eff86ffd640 10 monclient: dump: 2024-03-29T11:21:48.508 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:21:48.508 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:21:48.508 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:48.508 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:48.509 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:48.509 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:48.509 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:21:48.509 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:21:48.509 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:21:48.509 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:48.509 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.524+0000 7eff87fff640 4 set_mon_vals no callback set 2024-03-29T11:21:48.509 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.525+0000 7eff86ffd640 1 -- 172.21.15.17:0/2212135310 <== mon.2 v1:172.21.15.17:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7eff8c004f70 con 0x7eff980af900 2024-03-29T11:21:48.509 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.526+0000 7eff95d9b640 10 monclient: get_auth_request con 0x7eff78076680 auth_method 0 2024-03-29T11:21:48.509 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.526+0000 7eff86ffd640 10 client.4349.objecter ms_handle_connect 0x7eff78076680 2024-03-29T11:21:48.509 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.526+0000 7eff86ffd640 1 -- 172.21.15.17:0/2212135310 <== mon.2 v1:172.21.15.17:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7eff8c084b80 con 0x7eff980af900 2024-03-29T11:21:48.509 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.526+0000 7eff86ffd640 10 client.4349.objecter ms_dispatch 0x7eff98000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:48.510 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.526+0000 7eff86ffd640 3 client.4349.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:48.510 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.526+0000 7eff86ffd640 3 client.4349.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:48.510 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.526+0000 7eff86ffd640 20 client.4349.objecter dump_active .. 0 homeless 2024-03-29T11:21:48.510 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.527+0000 7eff9cbf4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:48.510 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.527+0000 7eff9cbf4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:48.510 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.527+0000 7eff9cbf4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:21:48.510 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.527+0000 7eff9cbf4640 1 -- 172.21.15.17:0/2212135310 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff7c005080 con 0x7eff980af900 2024-03-29T11:21:48.515 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.537+0000 7eff86ffd640 1 -- 172.21.15.17:0/2212135310 <== mon.2 v1:172.21.15.17:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7eff8c050460 con 0x7eff980af900 2024-03-29T11:21:48.515 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.537+0000 7eff86ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:48.515 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.537+0000 7eff86ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:48.835 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.858+0000 7eff9cbf4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:48.835 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.858+0000 7eff9cbf4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:48.835 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.858+0000 7eff9cbf4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:21:48.835 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.858+0000 7eff9cbf4640 1 -- 172.21.15.17:0/2212135310 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7eff7c005330 con 0x7eff980af900 2024-03-29T11:21:48.838 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.862+0000 7eff86ffd640 1 -- 172.21.15.17:0/2212135310 <== mon.2 v1:172.21.15.17:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7eff8c052390 con 0x7eff980af900 2024-03-29T11:21:48.839 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.862+0000 7eff86ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:48.839 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.862+0000 7eff86ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:48.839 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:21:48.839 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:48.842 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.866+0000 7eff9cbf4640 10 librados: watch_flush enter 2024-03-29T11:21:48.842 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.866+0000 7eff9cbf4640 10 librados: watch_flush exit 2024-03-29T11:21:48.842 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.866+0000 7eff9cbf4640 20 client.4349.objecter shutdown clearing up homeless session... 2024-03-29T11:21:48.842 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.866+0000 7eff9cbf4640 10 client.4349.objecter successfully canceled tick 2024-03-29T11:21:48.845 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.869+0000 7eff9cbf4640 1 -- 172.21.15.17:0/2212135310 >> v1:172.21.15.17:6816/3236185052 conn(0x7eff78076680 legacy=0x7eff78078b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:48.845 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.869+0000 7eff9cbf4640 10 monclient: shutdown 2024-03-29T11:21:48.845 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.869+0000 7eff9cbf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:48.845 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.869+0000 7eff9cbf4640 1 -- 172.21.15.17:0/2212135310 >> v1:172.21.15.17:6790/0 conn(0x7eff980af900 legacy=0x7eff981501c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:48.847 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.870+0000 7eff9cbf4640 1 -- 172.21.15.17:0/2212135310 shutdown_connections 2024-03-29T11:21:48.847 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.870+0000 7eff9cbf4640 1 -- 172.21.15.17:0/2212135310 >> 172.21.15.17:0/2212135310 conn(0x7eff980b35e0 msgr2=0x7eff980b07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:48.847 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.870+0000 7eff9cbf4640 1 -- 172.21.15.17:0/2212135310 shutdown_connections 2024-03-29T11:21:48.847 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.870+0000 7eff9cbf4640 1 -- 172.21.15.17:0/2212135310 wait complete. 2024-03-29T11:21:48.847 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:48.871+0000 7eff9cbf4640 1 librados: shutdown 2024-03-29T11:21:48.862 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:55.668 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:55.825 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.847+0000 7f6e262e0640 10 monclient: get_monmap_and_config 2024-03-29T11:21:55.825 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.847+0000 7f6e262e0640 10 monclient: build_initial_monmap 2024-03-29T11:21:55.825 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.848+0000 7f6e262e0640 10 monclient: monmap: 2024-03-29T11:21:55.825 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:21:55.825 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:21:55.826 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:21:55.849128+0000 2024-03-29T11:21:55.826 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:21:55.849128+0000 2024-03-29T11:21:55.826 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:55.826 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:55.826 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:21:55.826 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:21:55.826 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:21:55.826 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:55.829 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.851+0000 7f6e262e0640 1 Processor -- start 2024-03-29T11:21:55.829 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.851+0000 7f6e262e0640 1 -- start start 2024-03-29T11:21:55.829 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.851+0000 7f6e262e0640 10 monclient: init 2024-03-29T11:21:55.829 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.852+0000 7f6e262e0640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:55.829 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.852+0000 7f6e262e0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:21:55.829 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.852+0000 7f6e262e0640 10 monclient(hunting): picked mon.noname-c con 0x7f6e200a9930 addr v1:172.21.15.94:6789/0 2024-03-29T11:21:55.829 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.852+0000 7f6e262e0640 10 monclient(hunting): picked mon.noname-a con 0x7f6e20149dc0 addr v1:172.21.15.17:6789/0 2024-03-29T11:21:55.829 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.852+0000 7f6e262e0640 10 monclient(hunting): picked mon.noname-b con 0x7f6e2014ff40 addr v1:172.21.15.17:6790/0 2024-03-29T11:21:55.830 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.852+0000 7f6e262e0640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e200aae60 con 0x7f6e20149dc0 2024-03-29T11:21:55.830 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.852+0000 7f6e262e0640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e200b4310 con 0x7f6e200a9930 2024-03-29T11:21:55.830 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.852+0000 7f6e262e0640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e200afdd0 con 0x7f6e2014ff40 2024-03-29T11:21:55.830 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.852+0000 7f6e262e0640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:55.830 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.852+0000 7f6e262e0640 10 monclient(hunting): authenticate will time out at 1022.057090s 2024-03-29T11:21:55.831 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.855+0000 7f6e2487d640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f6e2014ff40 0x7f6e20152330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:52111/0 (socket says 172.21.15.17:52111) 2024-03-29T11:21:55.831 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.855+0000 7f6e2487d640 1 -- 172.21.15.17:0/1593264979 learned_addr learned my addr 172.21.15.17:0/1593264979 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:21:55.832 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.855+0000 7f6e1effd640 1 -- 172.21.15.17:0/1593264979 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 22613405 0 0) 0x7f6e200afdd0 con 0x7f6e2014ff40 2024-03-29T11:21:55.832 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.855+0000 7f6e1effd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:55.832 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.855+0000 7f6e1effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:55.832 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.855+0000 7f6e1effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:55.832 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.855+0000 7f6e1effd640 10 monclient(hunting): my global_id is 4352 2024-03-29T11:21:55.832 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.856+0000 7f6e1effd640 1 -- 172.21.15.17:0/1593264979 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e08003520 con 0x7f6e2014ff40 2024-03-29T11:21:55.832 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.856+0000 7f6e1effd640 1 -- 172.21.15.17:0/1593264979 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2828975833 0 0) 0x7f6e08003520 con 0x7f6e2014ff40 2024-03-29T11:21:55.833 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.856+0000 7f6e1effd640 1 -- 172.21.15.17:0/1593264979 >> v1:172.21.15.94:6789/0 conn(0x7f6e200a9930 legacy=0x7f6e20143a30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:21:55.833 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.856+0000 7f6e1effd640 1 -- 172.21.15.17:0/1593264979 >> v1:172.21.15.17:6789/0 conn(0x7f6e20149dc0 legacy=0x7f6e2014a240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:21:55.835 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.858+0000 7f6e1effd640 10 monclient: _finish_hunting 0 2024-03-29T11:21:55.835 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.858+0000 7f6e1effd640 1 monclient: found mon.noname-b 2024-03-29T11:21:55.836 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.858+0000 7f6e1effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:55.836 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.858+0000 7f6e1effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.17:6790/0 2024-03-29T11:21:55.836 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.858+0000 7f6e1effd640 1 -- 172.21.15.17:0/1593264979 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e2014a950 con 0x7f6e2014ff40 2024-03-29T11:21:55.836 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.858+0000 7f6e1effd640 10 monclient: _finish_auth 0 2024-03-29T11:21:55.836 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.858+0000 7f6e1effd640 10 monclient: _check_auth_tickets 2024-03-29T11:21:55.836 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.858+0000 7f6e1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:55.836 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.858+0000 7f6e262e0640 5 monclient: authenticate success, global_id 4352 2024-03-29T11:21:55.836 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.858+0000 7f6e262e0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:55.836 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.858+0000 7f6e1effd640 1 -- 172.21.15.17:0/1593264979 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f6e14001690 con 0x7f6e2014ff40 2024-03-29T11:21:55.836 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.858+0000 7f6e1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:55.836 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.859+0000 7f6e1effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:55.836 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.859+0000 7f6e1effd640 10 monclient: dump: 2024-03-29T11:21:55.836 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:21:55.836 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:21:55.836 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:55.837 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:55.837 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:55.837 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:55.837 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:21:55.837 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:21:55.837 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:21:55.837 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:55.837 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.859+0000 7f6e262e0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:55.837 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.859+0000 7f6e1effd640 1 -- 172.21.15.17:0/1593264979 <== mon.2 v1:172.21.15.17:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e14001920 con 0x7f6e2014ff40 2024-03-29T11:21:55.837 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.859+0000 7f6e1effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:55.838 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.859+0000 7f6e262e0640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:55.838 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.859+0000 7f6e262e0640 4 set_mon_vals no callback set 2024-03-29T11:21:55.838 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.860+0000 7f6e262e0640 10 monclient: shutdown 2024-03-29T11:21:55.838 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.860+0000 7f6e262e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:55.838 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.860+0000 7f6e262e0640 1 -- 172.21.15.17:0/1593264979 >> v1:172.21.15.17:6790/0 conn(0x7f6e2014ff40 legacy=0x7f6e20152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:55.839 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.860+0000 7f6e262e0640 1 -- 172.21.15.17:0/1593264979 shutdown_connections 2024-03-29T11:21:55.839 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.860+0000 7f6e262e0640 1 -- 172.21.15.17:0/1593264979 >> 172.21.15.17:0/1593264979 conn(0x7f6e200b35e0 msgr2=0x7f6e200b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:55.839 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.861+0000 7f6e262e0640 1 -- 172.21.15.17:0/1593264979 shutdown_connections 2024-03-29T11:21:55.842 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.864+0000 7f6e262e0640 1 -- 172.21.15.17:0/1593264979 wait complete. 2024-03-29T11:21:55.842 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.865+0000 7f6e262e0640 10 monclient: build_initial_monmap 2024-03-29T11:21:55.842 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.865+0000 7f6e262e0640 10 monclient: monmap: 2024-03-29T11:21:55.842 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:21:55.842 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:55.842 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:21:55.842 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:21:55.842 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:55.842 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:55.842 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:21:55.842 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:21:55.842 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:21:55.842 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:55.842 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.865+0000 7f6e262e0640 1 librados: starting msgr at 2024-03-29T11:21:55.843 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.865+0000 7f6e262e0640 1 librados: starting objecter 2024-03-29T11:21:55.843 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.865+0000 7f6e262e0640 1 Processor -- start 2024-03-29T11:21:55.849 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.866+0000 7f6e262e0640 1 -- start start 2024-03-29T11:21:55.849 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.866+0000 7f6e262e0640 1 librados: setting wanted keys 2024-03-29T11:21:55.849 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.866+0000 7f6e262e0640 1 librados: calling monclient init 2024-03-29T11:21:55.849 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.866+0000 7f6e262e0640 10 monclient: init 2024-03-29T11:21:55.849 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.866+0000 7f6e262e0640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:55.849 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.866+0000 7f6e262e0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:55.849 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.867+0000 7f6e262e0640 10 monclient(hunting): picked mon.noname-c con 0x7f6e200a9930 addr v1:172.21.15.17:6790/0 2024-03-29T11:21:55.849 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.867+0000 7f6e262e0640 10 monclient(hunting): picked mon.noname-b con 0x7f6e20149dc0 addr v1:172.21.15.94:6789/0 2024-03-29T11:21:55.849 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.867+0000 7f6e262e0640 10 monclient(hunting): picked mon.noname-a con 0x7f6e2014ff40 addr v1:172.21.15.17:6789/0 2024-03-29T11:21:55.849 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.867+0000 7f6e262e0640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e2006ed60 con 0x7f6e2014ff40 2024-03-29T11:21:55.849 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.867+0000 7f6e262e0640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e201db590 con 0x7f6e20149dc0 2024-03-29T11:21:55.850 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.867+0000 7f6e262e0640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e200ab380 con 0x7f6e200a9930 2024-03-29T11:21:55.850 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.867+0000 7f6e262e0640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:55.850 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.867+0000 7f6e262e0640 10 monclient(hunting): authenticate will time out at 1022.071960s 2024-03-29T11:21:55.850 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.871+0000 7f6e1ffff640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f6e200a9930 0x7f6e201dbab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:52209/0 (socket says 172.21.15.17:52209) 2024-03-29T11:21:55.850 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.871+0000 7f6e1ffff640 1 -- 172.21.15.17:0/2669976368 learned_addr learned my addr 172.21.15.17:0/2669976368 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:21:55.850 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.871+0000 7f6e1cff9640 10 client.?.objecter ms_handle_connect 0x7f6e200a9930 2024-03-29T11:21:55.850 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.871+0000 7f6e1cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:55.850 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.871+0000 7f6e1cff9640 1 -- 172.21.15.17:0/2669976368 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3632114267 0 0) 0x7f6e200ab380 con 0x7f6e200a9930 2024-03-29T11:21:55.850 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.871+0000 7f6e1cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:21:55.850 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.871+0000 7f6e1cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:55.850 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.871+0000 7f6e1cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:55.850 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.871+0000 7f6e1cff9640 10 monclient(hunting): my global_id is 4355 2024-03-29T11:21:55.850 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.871+0000 7f6e1cff9640 1 -- 172.21.15.17:0/2669976368 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e0c004eb0 con 0x7f6e200a9930 2024-03-29T11:21:55.850 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.871+0000 7f6e1cff9640 1 -- 172.21.15.17:0/2669976368 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2520819823 0 0) 0x7f6e0c004eb0 con 0x7f6e200a9930 2024-03-29T11:21:55.851 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.871+0000 7f6e1cff9640 1 -- 172.21.15.17:0/2669976368 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e200ab380 con 0x7f6e200a9930 2024-03-29T11:21:55.851 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.871+0000 7f6e1cff9640 1 -- 172.21.15.17:0/2669976368 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f6e10001870 con 0x7f6e200a9930 2024-03-29T11:21:55.851 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.871+0000 7f6e1cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:55.851 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.871+0000 7f6e1cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:55.851 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.871+0000 7f6e1cff9640 10 monclient(hunting): dump: 2024-03-29T11:21:55.851 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:21:55.851 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:21:55.851 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:55.851 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:55.851 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:55.851 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:55.851 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:21:55.851 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:21:55.851 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:21:55.851 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:55.852 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.871+0000 7f6e1cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:55.852 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.872+0000 7f6e1cff9640 1 -- 172.21.15.17:0/2669976368 <== mon.2 v1:172.21.15.17:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 413098754 0 0) 0x7f6e200ab380 con 0x7f6e200a9930 2024-03-29T11:21:55.852 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.872+0000 7f6e1cff9640 1 -- 172.21.15.17:0/2669976368 >> v1:172.21.15.94:6789/0 conn(0x7f6e20149dc0 legacy=0x7f6e2013efd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:21:55.852 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.874+0000 7f6e1cff9640 1 -- 172.21.15.17:0/2669976368 >> v1:172.21.15.17:6789/0 conn(0x7f6e2014ff40 legacy=0x7f6e201e5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:55.852 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.874+0000 7f6e1cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:21:55.852 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.874+0000 7f6e1cff9640 1 monclient: found mon.c 2024-03-29T11:21:55.853 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.874+0000 7f6e1cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:55.853 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.874+0000 7f6e1cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:21:55.853 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.874+0000 7f6e1cff9640 1 -- 172.21.15.17:0/2669976368 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e200ab510 con 0x7f6e200a9930 2024-03-29T11:21:55.853 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.874+0000 7f6e1cff9640 10 monclient: _finish_auth 0 2024-03-29T11:21:55.853 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.874+0000 7f6e1cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:21:55.853 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.874+0000 7f6e1cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:55.853 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.874+0000 7f6e262e0640 5 monclient: authenticate success, global_id 4355 2024-03-29T11:21:55.853 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.874+0000 7f6e262e0640 10 monclient: _renew_subs 2024-03-29T11:21:55.853 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.874+0000 7f6e262e0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:21:55.853 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.874+0000 7f6e262e0640 1 -- 172.21.15.17:0/2669976368 --> v1:172.21.15.17:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e201dc1c0 con 0x7f6e200a9930 2024-03-29T11:21:55.853 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.874+0000 7f6e262e0640 10 client.4355.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:55.853 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.874+0000 7f6e262e0640 10 monclient: _renew_subs 2024-03-29T11:21:55.853 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.874+0000 7f6e262e0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:21:55.853 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.874+0000 7f6e262e0640 1 -- 172.21.15.17:0/2669976368 --> v1:172.21.15.17:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e2013f8c0 con 0x7f6e200a9930 2024-03-29T11:21:55.854 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.874+0000 7f6e262e0640 1 librados: init done 2024-03-29T11:21:55.854 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.875+0000 7f6e1cff9640 1 -- 172.21.15.17:0/2669976368 <== mon.2 v1:172.21.15.17:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e100019d0 con 0x7f6e200a9930 2024-03-29T11:21:55.854 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.875+0000 7f6e1cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:55.854 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.875+0000 7f6e1cff9640 1 -- 172.21.15.17:0/2669976368 <== mon.2 v1:172.21.15.17:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f6e10004ae0 con 0x7f6e200a9930 2024-03-29T11:21:55.854 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.875+0000 7f6e1cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:55.854 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.875+0000 7f6e1cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:55.854 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.875+0000 7f6e1cff9640 10 monclient: dump: 2024-03-29T11:21:55.854 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:21:55.854 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:21:55.854 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:55.854 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:21:55.854 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:55.854 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:21:55.854 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:21:55.854 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:21:55.855 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:21:55.855 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:21:55.855 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.875+0000 7f6e1dffb640 4 set_mon_vals no callback set 2024-03-29T11:21:55.856 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.877+0000 7f6e1cff9640 1 -- 172.21.15.17:0/2669976368 <== mon.2 v1:172.21.15.17:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f6e1001aab0 con 0x7f6e200a9930 2024-03-29T11:21:55.856 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.878+0000 7f6e1cff9640 1 -- 172.21.15.17:0/2669976368 <== mon.2 v1:172.21.15.17:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f6e10084b60 con 0x7f6e200a9930 2024-03-29T11:21:55.856 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.878+0000 7f6e1cff9640 10 client.4355.objecter ms_dispatch 0x7f6e20000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:55.856 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.878+0000 7f6e1cff9640 3 client.4355.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:55.856 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.878+0000 7f6e1cff9640 3 client.4355.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:55.856 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.878+0000 7f6e1cff9640 20 client.4355.objecter dump_active .. 0 homeless 2024-03-29T11:21:55.856 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.878+0000 7f6e262e0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:55.856 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.878+0000 7f6e1f7fe640 10 monclient: get_auth_request con 0x7f6e0c06dc80 auth_method 0 2024-03-29T11:21:55.856 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.878+0000 7f6e262e0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:55.856 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.878+0000 7f6e262e0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:21:55.856 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.878+0000 7f6e262e0640 1 -- 172.21.15.17:0/2669976368 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e00005080 con 0x7f6e200a9930 2024-03-29T11:21:55.856 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.879+0000 7f6e1cff9640 10 client.4355.objecter ms_handle_connect 0x7f6e0c06dc80 2024-03-29T11:21:55.864 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.887+0000 7f6e1cff9640 1 -- 172.21.15.17:0/2669976368 <== mon.2 v1:172.21.15.17:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6e10050270 con 0x7f6e200a9930 2024-03-29T11:21:55.864 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.887+0000 7f6e1cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:55.864 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:55.887+0000 7f6e1cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:56.195 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:56.219+0000 7f6e262e0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:56.196 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:56.219+0000 7f6e262e0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:56.196 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:56.219+0000 7f6e262e0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:21:56.196 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:56.219+0000 7f6e262e0640 1 -- 172.21.15.17:0/2669976368 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e00005330 con 0x7f6e200a9930 2024-03-29T11:21:56.199 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:56.223+0000 7f6e1cff9640 1 -- 172.21.15.17:0/2669976368 <== mon.2 v1:172.21.15.17:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f6e10052370 con 0x7f6e200a9930 2024-03-29T11:21:56.199 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:56.223+0000 7f6e1cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:56.199 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:56.223+0000 7f6e1cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:56.199 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:21:56.199 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:56.202 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:56.226+0000 7f6e262e0640 10 librados: watch_flush enter 2024-03-29T11:21:56.203 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:56.227+0000 7f6e262e0640 10 librados: watch_flush exit 2024-03-29T11:21:56.203 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:56.227+0000 7f6e262e0640 20 client.4355.objecter shutdown clearing up homeless session... 2024-03-29T11:21:56.203 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:56.227+0000 7f6e262e0640 10 client.4355.objecter successfully canceled tick 2024-03-29T11:21:56.203 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:56.227+0000 7f6e262e0640 1 -- 172.21.15.17:0/2669976368 >> v1:172.21.15.17:6816/3236185052 conn(0x7f6e0c06dc80 legacy=0x7f6e0c070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:56.203 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:56.227+0000 7f6e262e0640 10 monclient: shutdown 2024-03-29T11:21:56.203 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:56.227+0000 7f6e262e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:56.203 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:56.227+0000 7f6e262e0640 1 -- 172.21.15.17:0/2669976368 >> v1:172.21.15.17:6790/0 conn(0x7f6e200a9930 legacy=0x7f6e201dbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:56.204 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:56.227+0000 7f6e262e0640 1 -- 172.21.15.17:0/2669976368 shutdown_connections 2024-03-29T11:21:56.204 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:56.227+0000 7f6e262e0640 1 -- 172.21.15.17:0/2669976368 >> 172.21.15.17:0/2669976368 conn(0x7f6e200b35e0 msgr2=0x7f6e200b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:56.205 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:56.229+0000 7f6e262e0640 1 -- 172.21.15.17:0/2669976368 shutdown_connections 2024-03-29T11:21:56.205 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:56.229+0000 7f6e262e0640 1 -- 172.21.15.17:0/2669976368 wait complete. 2024-03-29T11:21:56.206 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:21:56.230+0000 7f6e262e0640 1 librados: shutdown 2024-03-29T11:21:56.223 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:03.029 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:03.179 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.199+0000 7f4134513640 10 monclient: get_monmap_and_config 2024-03-29T11:22:03.179 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.199+0000 7f4134513640 10 monclient: build_initial_monmap 2024-03-29T11:22:03.179 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.199+0000 7f4134513640 10 monclient: monmap: 2024-03-29T11:22:03.179 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:22:03.179 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:22:03.179 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:22:03.200345+0000 2024-03-29T11:22:03.179 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:22:03.200345+0000 2024-03-29T11:22:03.179 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:03.179 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:03.180 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:22:03.180 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:22:03.180 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:22:03.180 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:03.180 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.202+0000 7f4134513640 1 Processor -- start 2024-03-29T11:22:03.180 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.202+0000 7f4134513640 1 -- start start 2024-03-29T11:22:03.180 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.202+0000 7f4134513640 10 monclient: init 2024-03-29T11:22:03.184 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.203+0000 7f4134513640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:03.184 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.203+0000 7f4134513640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:03.184 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.203+0000 7f4134513640 10 monclient(hunting): picked mon.noname-b con 0x7f412c0b4100 addr v1:172.21.15.17:6790/0 2024-03-29T11:22:03.184 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.203+0000 7f4134513640 10 monclient(hunting): picked mon.noname-c con 0x7f412c13ee90 addr v1:172.21.15.94:6789/0 2024-03-29T11:22:03.185 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.203+0000 7f4134513640 10 monclient(hunting): picked mon.noname-a con 0x7f412c0af900 addr v1:172.21.15.17:6789/0 2024-03-29T11:22:03.185 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.203+0000 7f4134513640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f412c0afd10 con 0x7f412c0af900 2024-03-29T11:22:03.185 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.203+0000 7f4134513640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f412c0af6f0 con 0x7f412c13ee90 2024-03-29T11:22:03.185 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.203+0000 7f4134513640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f412c0aaf60 con 0x7f412c0b4100 2024-03-29T11:22:03.185 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.203+0000 7f4134513640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:03.185 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.203+0000 7f4134513640 10 monclient(hunting): authenticate will time out at 1029.408670s 2024-03-29T11:22:03.185 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.207+0000 7f4131aae640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f412c13ee90 0x7f412c0adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:37323/0 (socket says 172.21.15.17:37323) 2024-03-29T11:22:03.185 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.207+0000 7f4131aae640 1 -- 172.21.15.17:0/496101844 learned_addr learned my addr 172.21.15.17:0/496101844 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:22:03.191 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.208+0000 7f41312ad640 1 -- 172.21.15.17:0/496101844 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1652002916 0 0) 0x7f412c0af6f0 con 0x7f412c13ee90 2024-03-29T11:22:03.191 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.208+0000 7f41312ad640 10 monclient(hunting): _negotiate 2024-03-29T11:22:03.191 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.208+0000 7f41312ad640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:03.191 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.208+0000 7f41312ad640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:03.191 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.208+0000 7f41312ad640 10 monclient(hunting): my global_id is 4336 2024-03-29T11:22:03.191 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.208+0000 7f41312ad640 1 -- 172.21.15.17:0/496101844 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4118003520 con 0x7f412c13ee90 2024-03-29T11:22:03.192 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.208+0000 7f41312ad640 1 -- 172.21.15.17:0/496101844 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4149406369 0 0) 0x7f412c0aaf60 con 0x7f412c0b4100 2024-03-29T11:22:03.192 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.208+0000 7f41312ad640 10 monclient(hunting): _negotiate 2024-03-29T11:22:03.192 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.208+0000 7f41312ad640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:03.192 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.208+0000 7f41312ad640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:03.192 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.208+0000 7f41312ad640 10 monclient(hunting): my global_id is 4358 2024-03-29T11:22:03.192 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.208+0000 7f41312ad640 1 -- 172.21.15.17:0/496101844 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41180048a0 con 0x7f412c0b4100 2024-03-29T11:22:03.192 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.209+0000 7f41312ad640 1 -- 172.21.15.17:0/496101844 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 171841254 0 0) 0x7f41180048a0 con 0x7f412c0b4100 2024-03-29T11:22:03.192 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.210+0000 7f41312ad640 1 -- 172.21.15.17:0/496101844 >> v1:172.21.15.94:6789/0 conn(0x7f412c13ee90 legacy=0x7f412c0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:03.192 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.210+0000 7f41312ad640 1 -- 172.21.15.17:0/496101844 >> v1:172.21.15.17:6789/0 conn(0x7f412c0af900 legacy=0x7f412c13b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:03.192 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.211+0000 7f41312ad640 10 monclient: _finish_hunting 0 2024-03-29T11:22:03.192 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.211+0000 7f41312ad640 1 monclient: found mon.noname-b 2024-03-29T11:22:03.192 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.211+0000 7f41312ad640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:03.192 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.211+0000 7f41312ad640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.17:6790/0 2024-03-29T11:22:03.193 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.211+0000 7f41312ad640 1 -- 172.21.15.17:0/496101844 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f412c13e970 con 0x7f412c0b4100 2024-03-29T11:22:03.193 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.211+0000 7f41312ad640 10 monclient: _finish_auth 0 2024-03-29T11:22:03.193 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.211+0000 7f41312ad640 10 monclient: _check_auth_tickets 2024-03-29T11:22:03.193 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.211+0000 7f41312ad640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:03.193 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.211+0000 7f41312ad640 1 -- 172.21.15.17:0/496101844 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f41240019b0 con 0x7f412c0b4100 2024-03-29T11:22:03.193 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.211+0000 7f41312ad640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:03.193 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.211+0000 7f41312ad640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:03.193 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.211+0000 7f41312ad640 10 monclient: dump: 2024-03-29T11:22:03.193 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:22:03.193 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:22:03.193 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:03.193 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:03.193 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:03.193 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:03.193 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:22:03.194 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:22:03.194 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:22:03.194 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:03.194 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.211+0000 7f4134513640 5 monclient: authenticate success, global_id 4358 2024-03-29T11:22:03.194 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.211+0000 7f4134513640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:03.194 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.211+0000 7f41312ad640 1 -- 172.21.15.17:0/496101844 <== mon.2 v1:172.21.15.17:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4124001c00 con 0x7f412c0b4100 2024-03-29T11:22:03.194 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.211+0000 7f41312ad640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:03.194 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.211+0000 7f41312ad640 1 -- 172.21.15.17:0/496101844 <== mon.2 v1:172.21.15.17:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f4124001f90 con 0x7f412c0b4100 2024-03-29T11:22:03.194 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.211+0000 7f41312ad640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:03.194 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.211+0000 7f41312ad640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:03.194 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.211+0000 7f41312ad640 10 monclient: dump: 2024-03-29T11:22:03.194 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:22:03.194 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:22:03.194 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:03.194 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:03.195 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:03.195 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:03.195 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:22:03.195 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:22:03.195 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:22:03.195 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:03.195 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.211+0000 7f4134513640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:03.195 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.211+0000 7f4134513640 4 set_mon_vals no callback set 2024-03-29T11:22:03.195 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.212+0000 7f4134513640 10 monclient: shutdown 2024-03-29T11:22:03.195 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.212+0000 7f4134513640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:03.195 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.212+0000 7f4134513640 1 -- 172.21.15.17:0/496101844 >> v1:172.21.15.17:6790/0 conn(0x7f412c0b4100 legacy=0x7f412c0a9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:03.195 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.215+0000 7f4134513640 1 -- 172.21.15.17:0/496101844 shutdown_connections 2024-03-29T11:22:03.195 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.215+0000 7f4134513640 1 -- 172.21.15.17:0/496101844 >> 172.21.15.17:0/496101844 conn(0x7f412c0b35e0 msgr2=0x7f412c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:03.197 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.216+0000 7f4134513640 1 -- 172.21.15.17:0/496101844 shutdown_connections 2024-03-29T11:22:03.197 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.216+0000 7f4134513640 1 -- 172.21.15.17:0/496101844 wait complete. 2024-03-29T11:22:03.197 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.217+0000 7f4134513640 10 monclient: build_initial_monmap 2024-03-29T11:22:03.197 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.217+0000 7f4134513640 10 monclient: monmap: 2024-03-29T11:22:03.197 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:22:03.197 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:03.197 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:22:03.197 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:22:03.197 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:03.198 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:03.198 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:22:03.198 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:22:03.198 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:22:03.198 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:03.198 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.217+0000 7f4134513640 1 librados: starting msgr at 2024-03-29T11:22:03.198 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.217+0000 7f4134513640 1 librados: starting objecter 2024-03-29T11:22:03.198 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.218+0000 7f4134513640 1 Processor -- start 2024-03-29T11:22:03.198 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.218+0000 7f4134513640 1 -- start start 2024-03-29T11:22:03.198 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.218+0000 7f4134513640 1 librados: setting wanted keys 2024-03-29T11:22:03.198 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.218+0000 7f4134513640 1 librados: calling monclient init 2024-03-29T11:22:03.198 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.218+0000 7f4134513640 10 monclient: init 2024-03-29T11:22:03.198 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.219+0000 7f4134513640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:03.198 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.219+0000 7f4134513640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:22:03.198 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.219+0000 7f4134513640 10 monclient(hunting): picked mon.noname-a con 0x7f412c0af900 addr v1:172.21.15.17:6789/0 2024-03-29T11:22:03.199 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.222+0000 7f4134513640 10 monclient(hunting): picked mon.noname-c con 0x7f412c0b4100 addr v1:172.21.15.17:6790/0 2024-03-29T11:22:03.200 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.222+0000 7f4131aae640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f412c0b4100 0x7f412c1d6950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:56333/0 (socket says 172.21.15.17:56333) 2024-03-29T11:22:03.200 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.222+0000 7f4131aae640 1 -- 172.21.15.17:0/2455729549 learned_addr learned my addr 172.21.15.17:0/2455729549 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:22:03.200 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.222+0000 7f4134513640 10 monclient(hunting): picked mon.noname-b con 0x7f412c13ee90 addr v1:172.21.15.94:6789/0 2024-03-29T11:22:03.200 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.222+0000 7f4134513640 1 -- 172.21.15.17:0/2455729549 --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f412c06e860 con 0x7f412c0af900 2024-03-29T11:22:03.200 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.222+0000 7f4134513640 1 -- 172.21.15.17:0/2455729549 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f412c0707a0 con 0x7f412c13ee90 2024-03-29T11:22:03.200 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.222+0000 7f4134513640 1 -- 172.21.15.17:0/2455729549 --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f412c06ef10 con 0x7f412c0b4100 2024-03-29T11:22:03.200 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.222+0000 7f4134513640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:03.200 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.222+0000 7f4134513640 10 monclient(hunting): authenticate will time out at 1029.427825s 2024-03-29T11:22:03.201 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.224+0000 7f4122ffd640 10 client.?.objecter ms_handle_connect 0x7f412c0b4100 2024-03-29T11:22:03.201 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.224+0000 7f4122ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:03.204 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.227+0000 7f4122ffd640 1 -- 172.21.15.17:0/2455729549 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4146180964 0 0) 0x7f412c06ef10 con 0x7f412c0b4100 2024-03-29T11:22:03.205 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.227+0000 7f4122ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:03.205 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.227+0000 7f4122ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:03.205 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.227+0000 7f4122ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:03.205 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.227+0000 7f4122ffd640 10 monclient(hunting): my global_id is 4361 2024-03-29T11:22:03.205 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.227+0000 7f4122ffd640 1 -- 172.21.15.17:0/2455729549 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4110004eb0 con 0x7f412c0b4100 2024-03-29T11:22:03.205 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.227+0000 7f4122ffd640 10 client.?.objecter ms_handle_connect 0x7f412c0af900 2024-03-29T11:22:03.205 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.227+0000 7f4122ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:03.208 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.228+0000 7f4122ffd640 1 -- 172.21.15.17:0/2455729549 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1357524932 0 0) 0x7f412c06e860 con 0x7f412c0af900 2024-03-29T11:22:03.208 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.229+0000 7f4122ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:03.208 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.229+0000 7f4122ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:03.208 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.229+0000 7f4122ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:03.208 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.229+0000 7f4122ffd640 10 monclient(hunting): my global_id is 4338 2024-03-29T11:22:03.208 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.229+0000 7f4122ffd640 1 -- 172.21.15.17:0/2455729549 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f412c06ef10 con 0x7f412c0af900 2024-03-29T11:22:03.208 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.229+0000 7f4122ffd640 1 -- 172.21.15.17:0/2455729549 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4269943452 0 0) 0x7f4110004eb0 con 0x7f412c0b4100 2024-03-29T11:22:03.208 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.229+0000 7f4122ffd640 1 -- 172.21.15.17:0/2455729549 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f412c06e860 con 0x7f412c0b4100 2024-03-29T11:22:03.208 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.229+0000 7f4122ffd640 1 -- 172.21.15.17:0/2455729549 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f4128001870 con 0x7f412c0b4100 2024-03-29T11:22:03.208 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.229+0000 7f4122ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:03.208 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.229+0000 7f4122ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:03.208 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.229+0000 7f4122ffd640 10 monclient(hunting): dump: 2024-03-29T11:22:03.209 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:22:03.209 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:22:03.209 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:03.209 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:03.209 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:03.209 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:03.209 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:22:03.209 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:22:03.209 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:22:03.209 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:03.209 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.229+0000 7f4122ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:03.209 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.229+0000 7f4122ffd640 10 client.?.objecter ms_handle_connect 0x7f412c13ee90 2024-03-29T11:22:03.209 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.229+0000 7f4122ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:03.209 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.229+0000 7f4122ffd640 1 -- 172.21.15.17:0/2455729549 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1185415268 0 0) 0x7f412c0707a0 con 0x7f412c13ee90 2024-03-29T11:22:03.210 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.229+0000 7f4122ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:03.210 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.229+0000 7f4122ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:03.210 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.229+0000 7f4122ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:03.210 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.229+0000 7f4122ffd640 10 monclient(hunting): my global_id is 4339 2024-03-29T11:22:03.210 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.229+0000 7f4122ffd640 1 -- 172.21.15.17:0/2455729549 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4110004eb0 con 0x7f412c13ee90 2024-03-29T11:22:03.210 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.231+0000 7f4122ffd640 1 -- 172.21.15.17:0/2455729549 <== mon.2 v1:172.21.15.17:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 29760173 0 0) 0x7f412c06e860 con 0x7f412c0b4100 2024-03-29T11:22:03.210 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.231+0000 7f4122ffd640 1 -- 172.21.15.17:0/2455729549 >> v1:172.21.15.94:6789/0 conn(0x7f412c13ee90 legacy=0x7f412c1d7060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:03.210 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.231+0000 7f4122ffd640 1 -- 172.21.15.17:0/2455729549 >> v1:172.21.15.17:6789/0 conn(0x7f412c0af900 legacy=0x7f412c1376b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:03.210 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.231+0000 7f4122ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:22:03.210 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.231+0000 7f4122ffd640 1 monclient: found mon.c 2024-03-29T11:22:03.210 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.231+0000 7f4122ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:03.210 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.231+0000 7f4122ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:22:03.210 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.231+0000 7f4122ffd640 1 -- 172.21.15.17:0/2455729549 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f412c06f6f0 con 0x7f412c0b4100 2024-03-29T11:22:03.211 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.231+0000 7f4122ffd640 10 monclient: _finish_auth 0 2024-03-29T11:22:03.211 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.231+0000 7f4122ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:22:03.211 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.231+0000 7f4122ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:03.211 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.231+0000 7f4134513640 5 monclient: authenticate success, global_id 4361 2024-03-29T11:22:03.211 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.231+0000 7f4134513640 10 monclient: _renew_subs 2024-03-29T11:22:03.211 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.231+0000 7f4134513640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:22:03.211 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.231+0000 7f4134513640 1 -- 172.21.15.17:0/2455729549 --> v1:172.21.15.17:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f412c13fa00 con 0x7f412c0b4100 2024-03-29T11:22:03.211 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.231+0000 7f4134513640 10 client.4361.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:03.211 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.231+0000 7f4134513640 10 monclient: _renew_subs 2024-03-29T11:22:03.211 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.231+0000 7f4134513640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:22:03.211 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.231+0000 7f4134513640 1 -- 172.21.15.17:0/2455729549 --> v1:172.21.15.17:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f412c1424e0 con 0x7f412c0b4100 2024-03-29T11:22:03.211 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.231+0000 7f4134513640 1 librados: init done 2024-03-29T11:22:03.212 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.233+0000 7f4134513640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:03.212 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.233+0000 7f4134513640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:03.212 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.234+0000 7f4134513640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:22:03.212 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.234+0000 7f4134513640 1 -- 172.21.15.17:0/2455729549 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f412c06f8e0 con 0x7f412c0b4100 2024-03-29T11:22:03.213 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.236+0000 7f4122ffd640 1 -- 172.21.15.17:0/2455729549 <== mon.2 v1:172.21.15.17:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4128001d60 con 0x7f412c0b4100 2024-03-29T11:22:03.213 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.236+0000 7f4122ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:03.213 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.236+0000 7f4122ffd640 1 -- 172.21.15.17:0/2455729549 <== mon.2 v1:172.21.15.17:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f41280049c0 con 0x7f412c0b4100 2024-03-29T11:22:03.213 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.236+0000 7f4122ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:03.213 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.236+0000 7f4122ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:03.213 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.236+0000 7f4122ffd640 10 monclient: dump: 2024-03-29T11:22:03.213 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:22:03.214 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:22:03.214 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:03.214 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:03.214 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:03.214 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:03.214 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:22:03.214 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:22:03.214 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:22:03.214 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:03.214 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.236+0000 7f4130aac640 4 set_mon_vals no callback set 2024-03-29T11:22:03.215 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.237+0000 7f4122ffd640 1 -- 172.21.15.17:0/2455729549 <== mon.2 v1:172.21.15.17:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f412801aa50 con 0x7f412c0b4100 2024-03-29T11:22:03.215 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.238+0000 7f4122ffd640 1 -- 172.21.15.17:0/2455729549 <== mon.2 v1:172.21.15.17:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f41280849f0 con 0x7f412c0b4100 2024-03-29T11:22:03.215 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.238+0000 7f4122ffd640 10 client.4361.objecter ms_dispatch 0x7f412c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:03.215 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.238+0000 7f4122ffd640 3 client.4361.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:03.215 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.238+0000 7f4122ffd640 3 client.4361.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:03.215 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.238+0000 7f4122ffd640 20 client.4361.objecter dump_active .. 0 homeless 2024-03-29T11:22:03.215 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.238+0000 7f41322af640 10 monclient: get_auth_request con 0x7f411006dc40 auth_method 0 2024-03-29T11:22:03.215 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.239+0000 7f4122ffd640 10 client.4361.objecter ms_handle_connect 0x7f411006dc40 2024-03-29T11:22:03.223 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.247+0000 7f4122ffd640 1 -- 172.21.15.17:0/2455729549 <== mon.2 v1:172.21.15.17:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f41280502d0 con 0x7f412c0b4100 2024-03-29T11:22:03.224 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.247+0000 7f4122ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:03.224 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.247+0000 7f4122ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:03.554 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.578+0000 7f4134513640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:03.555 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.578+0000 7f4134513640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:03.555 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.578+0000 7f4134513640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:22:03.555 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.578+0000 7f4134513640 1 -- 172.21.15.17:0/2455729549 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f412c075f10 con 0x7f412c0b4100 2024-03-29T11:22:03.559 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.583+0000 7f4122ffd640 1 -- 172.21.15.17:0/2455729549 <== mon.2 v1:172.21.15.17:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f4128052200 con 0x7f412c0b4100 2024-03-29T11:22:03.559 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.583+0000 7f4122ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:03.559 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.583+0000 7f4122ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:03.560 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:22:03.560 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:03.563 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.585+0000 7f4134513640 10 librados: watch_flush enter 2024-03-29T11:22:03.563 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.586+0000 7f4134513640 10 librados: watch_flush exit 2024-03-29T11:22:03.563 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.586+0000 7f4134513640 20 client.4361.objecter shutdown clearing up homeless session... 2024-03-29T11:22:03.563 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.586+0000 7f4134513640 10 client.4361.objecter successfully canceled tick 2024-03-29T11:22:03.563 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.586+0000 7f4134513640 1 -- 172.21.15.17:0/2455729549 >> v1:172.21.15.17:6816/3236185052 conn(0x7f411006dc40 legacy=0x7f4110070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:03.563 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.586+0000 7f4134513640 10 monclient: shutdown 2024-03-29T11:22:03.563 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.586+0000 7f4134513640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:03.563 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.586+0000 7f4134513640 1 -- 172.21.15.17:0/2455729549 >> v1:172.21.15.17:6790/0 conn(0x7f412c0b4100 legacy=0x7f412c1d6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:03.563 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.586+0000 7f4134513640 1 -- 172.21.15.17:0/2455729549 shutdown_connections 2024-03-29T11:22:03.563 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.586+0000 7f4134513640 1 -- 172.21.15.17:0/2455729549 >> 172.21.15.17:0/2455729549 conn(0x7f412c0b35e0 msgr2=0x7f412c0b11b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:03.563 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.586+0000 7f4134513640 1 -- 172.21.15.17:0/2455729549 shutdown_connections 2024-03-29T11:22:03.563 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.586+0000 7f4134513640 1 -- 172.21.15.17:0/2455729549 wait complete. 2024-03-29T11:22:03.563 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:03.586+0000 7f4134513640 1 librados: shutdown 2024-03-29T11:22:03.582 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:10.390 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:10.541 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.565+0000 7f5f2e33e640 10 monclient: get_monmap_and_config 2024-03-29T11:22:10.541 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.565+0000 7f5f2e33e640 10 monclient: build_initial_monmap 2024-03-29T11:22:10.541 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.565+0000 7f5f2e33e640 10 monclient: monmap: 2024-03-29T11:22:10.541 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:22:10.541 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:22:10.541 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:22:10.566074+0000 2024-03-29T11:22:10.542 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:22:10.566074+0000 2024-03-29T11:22:10.542 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:10.542 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:10.542 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:22:10.542 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:22:10.542 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:22:10.542 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:10.545 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.569+0000 7f5f2e33e640 1 Processor -- start 2024-03-29T11:22:10.545 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.569+0000 7f5f2e33e640 1 -- start start 2024-03-29T11:22:10.545 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.569+0000 7f5f2e33e640 10 monclient: init 2024-03-29T11:22:10.550 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.570+0000 7f5f2e33e640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:10.550 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.570+0000 7f5f2e33e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:22:10.550 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.570+0000 7f5f2e33e640 10 monclient(hunting): picked mon.noname-b con 0x7f5f281457f0 addr v1:172.21.15.17:6790/0 2024-03-29T11:22:10.550 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.570+0000 7f5f2e33e640 10 monclient(hunting): picked mon.noname-a con 0x7f5f2813f650 addr v1:172.21.15.17:6789/0 2024-03-29T11:22:10.550 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.570+0000 7f5f2e33e640 10 monclient(hunting): picked mon.noname-c con 0x7f5f280a9d70 addr v1:172.21.15.94:6789/0 2024-03-29T11:22:10.550 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.570+0000 7f5f2e33e640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f280aaef0 con 0x7f5f2813f650 2024-03-29T11:22:10.550 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.570+0000 7f5f2e33e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f28145bd0 con 0x7f5f280a9d70 2024-03-29T11:22:10.550 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.570+0000 7f5f2e33e640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f28145eb0 con 0x7f5f281457f0 2024-03-29T11:22:10.550 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.570+0000 7f5f2e33e640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:10.551 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.570+0000 7f5f2e33e640 10 monclient(hunting): authenticate will time out at 1036.775461s 2024-03-29T11:22:10.551 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.574+0000 7f5f277fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f5f2813f650 0x7f5f280b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:36331/0 (socket says 172.21.15.17:36331) 2024-03-29T11:22:10.551 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.574+0000 7f5f277fe640 1 -- 172.21.15.17:0/355460633 learned_addr learned my addr 172.21.15.17:0/355460633 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:22:10.557 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.580+0000 7f5f26ffd640 1 -- 172.21.15.17:0/355460633 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3352691348 0 0) 0x7f5f280aaef0 con 0x7f5f2813f650 2024-03-29T11:22:10.557 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.580+0000 7f5f26ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:10.558 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.580+0000 7f5f26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:10.558 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.580+0000 7f5f26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:10.558 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.580+0000 7f5f26ffd640 10 monclient(hunting): my global_id is 4341 2024-03-29T11:22:10.558 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.580+0000 7f5f26ffd640 1 -- 172.21.15.17:0/355460633 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f10003520 con 0x7f5f2813f650 2024-03-29T11:22:10.558 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.580+0000 7f5f26ffd640 1 -- 172.21.15.17:0/355460633 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 923851711 0 0) 0x7f5f28145bd0 con 0x7f5f280a9d70 2024-03-29T11:22:10.558 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.580+0000 7f5f26ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:10.558 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.580+0000 7f5f26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:10.558 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.580+0000 7f5f26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:10.558 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.580+0000 7f5f26ffd640 10 monclient(hunting): my global_id is 4342 2024-03-29T11:22:10.558 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.580+0000 7f5f26ffd640 1 -- 172.21.15.17:0/355460633 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f280aaef0 con 0x7f5f280a9d70 2024-03-29T11:22:10.558 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.580+0000 7f5f26ffd640 1 -- 172.21.15.17:0/355460633 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 960523884 0 0) 0x7f5f28145eb0 con 0x7f5f281457f0 2024-03-29T11:22:10.558 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.580+0000 7f5f26ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:10.558 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.580+0000 7f5f26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:10.558 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.580+0000 7f5f26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:10.559 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.580+0000 7f5f26ffd640 10 monclient(hunting): my global_id is 4364 2024-03-29T11:22:10.559 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.580+0000 7f5f26ffd640 1 -- 172.21.15.17:0/355460633 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f28145bd0 con 0x7f5f281457f0 2024-03-29T11:22:10.559 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.581+0000 7f5f26ffd640 1 -- 172.21.15.17:0/355460633 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1793957365 0 0) 0x7f5f28145bd0 con 0x7f5f281457f0 2024-03-29T11:22:10.559 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.581+0000 7f5f26ffd640 1 -- 172.21.15.17:0/355460633 >> v1:172.21.15.94:6789/0 conn(0x7f5f280a9d70 legacy=0x7f5f2814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:10.559 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.581+0000 7f5f26ffd640 1 -- 172.21.15.17:0/355460633 >> v1:172.21.15.17:6789/0 conn(0x7f5f2813f650 legacy=0x7f5f280b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:10.559 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.581+0000 7f5f26ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:22:10.559 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.581+0000 7f5f26ffd640 1 monclient: found mon.noname-b 2024-03-29T11:22:10.559 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.581+0000 7f5f26ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:10.559 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.581+0000 7f5f26ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.17:6790/0 2024-03-29T11:22:10.559 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.581+0000 7f5f26ffd640 1 -- 172.21.15.17:0/355460633 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f28146340 con 0x7f5f281457f0 2024-03-29T11:22:10.559 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.581+0000 7f5f26ffd640 10 monclient: _finish_auth 0 2024-03-29T11:22:10.559 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.581+0000 7f5f26ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:22:10.559 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.581+0000 7f5f26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:10.559 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.581+0000 7f5f2e33e640 5 monclient: authenticate success, global_id 4364 2024-03-29T11:22:10.559 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.581+0000 7f5f2e33e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:10.560 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.581+0000 7f5f26ffd640 1 -- 172.21.15.17:0/355460633 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f5f1c001460 con 0x7f5f281457f0 2024-03-29T11:22:10.560 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.581+0000 7f5f26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:10.560 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.581+0000 7f5f26ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:10.560 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.581+0000 7f5f26ffd640 10 monclient: dump: 2024-03-29T11:22:10.560 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:22:10.560 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:22:10.560 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:10.560 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:10.560 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:10.560 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:10.560 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:22:10.560 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:22:10.560 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:22:10.560 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:10.560 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.581+0000 7f5f2e33e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:10.561 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.581+0000 7f5f26ffd640 1 -- 172.21.15.17:0/355460633 <== mon.2 v1:172.21.15.17:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f1c0015c0 con 0x7f5f281457f0 2024-03-29T11:22:10.561 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.581+0000 7f5f26ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:10.561 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.581+0000 7f5f2e33e640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:10.561 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.581+0000 7f5f2e33e640 4 set_mon_vals no callback set 2024-03-29T11:22:10.561 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.581+0000 7f5f26ffd640 1 -- 172.21.15.17:0/355460633 <== mon.2 v1:172.21.15.17:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f5f1c001a10 con 0x7f5f281457f0 2024-03-29T11:22:10.561 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.581+0000 7f5f26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:10.561 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.581+0000 7f5f26ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:10.561 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.581+0000 7f5f26ffd640 10 monclient: dump: 2024-03-29T11:22:10.561 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:22:10.561 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:22:10.561 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:10.561 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:10.561 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:10.561 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:10.561 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:22:10.562 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:22:10.562 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:22:10.562 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:10.564 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.582+0000 7f5f2e33e640 10 monclient: shutdown 2024-03-29T11:22:10.564 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.582+0000 7f5f2e33e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:10.564 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.582+0000 7f5f2e33e640 1 -- 172.21.15.17:0/355460633 >> v1:172.21.15.17:6790/0 conn(0x7f5f281457f0 legacy=0x7f5f2813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:10.564 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.582+0000 7f5f2e33e640 1 -- 172.21.15.17:0/355460633 shutdown_connections 2024-03-29T11:22:10.564 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.582+0000 7f5f2e33e640 1 -- 172.21.15.17:0/355460633 >> 172.21.15.17:0/355460633 conn(0x7f5f28066a00 msgr2=0x7f5f28066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:10.564 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.582+0000 7f5f2e33e640 1 -- 172.21.15.17:0/355460633 shutdown_connections 2024-03-29T11:22:10.564 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.582+0000 7f5f2e33e640 1 -- 172.21.15.17:0/355460633 wait complete. 2024-03-29T11:22:10.564 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.583+0000 7f5f2e33e640 10 monclient: build_initial_monmap 2024-03-29T11:22:10.565 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.583+0000 7f5f2e33e640 10 monclient: monmap: 2024-03-29T11:22:10.565 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:22:10.565 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:10.565 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:22:10.565 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:22:10.565 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:10.565 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:10.565 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:22:10.565 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:22:10.565 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:22:10.565 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:10.565 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.584+0000 7f5f2e33e640 1 librados: starting msgr at 2024-03-29T11:22:10.565 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.584+0000 7f5f2e33e640 1 librados: starting objecter 2024-03-29T11:22:10.565 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.584+0000 7f5f2e33e640 1 Processor -- start 2024-03-29T11:22:10.565 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.585+0000 7f5f2e33e640 1 -- start start 2024-03-29T11:22:10.565 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.585+0000 7f5f2e33e640 1 librados: setting wanted keys 2024-03-29T11:22:10.566 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.585+0000 7f5f2e33e640 1 librados: calling monclient init 2024-03-29T11:22:10.566 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.585+0000 7f5f2e33e640 10 monclient: init 2024-03-29T11:22:10.566 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.586+0000 7f5f2e33e640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:10.566 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.586+0000 7f5f2e33e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:22:10.566 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.586+0000 7f5f2e33e640 10 monclient(hunting): picked mon.noname-c con 0x7f5f280a9d70 addr v1:172.21.15.17:6790/0 2024-03-29T11:22:10.566 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.586+0000 7f5f2e33e640 10 monclient(hunting): picked mon.noname-a con 0x7f5f2813f650 addr v1:172.21.15.17:6789/0 2024-03-29T11:22:10.566 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.586+0000 7f5f2e33e640 10 monclient(hunting): picked mon.noname-b con 0x7f5f281457f0 addr v1:172.21.15.94:6789/0 2024-03-29T11:22:10.566 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.586+0000 7f5f2e33e640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f28142650 con 0x7f5f2813f650 2024-03-29T11:22:10.566 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.586+0000 7f5f2e33e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f2806eb00 con 0x7f5f281457f0 2024-03-29T11:22:10.566 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.586+0000 7f5f2e33e640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f281dc260 con 0x7f5f280a9d70 2024-03-29T11:22:10.566 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.586+0000 7f5f2e33e640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:10.566 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.586+0000 7f5f2e33e640 10 monclient(hunting): authenticate will time out at 1036.791102s 2024-03-29T11:22:10.567 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.590+0000 7f5f2c8db640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f5f281457f0 0x7f5f281dbb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:47787/0 (socket says 172.21.15.17:47787) 2024-03-29T11:22:10.567 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.590+0000 7f5f2c8db640 1 -- 172.21.15.17:0/698292028 learned_addr learned my addr 172.21.15.17:0/698292028 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:22:10.567 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.591+0000 7f5f24ff9640 10 client.?.objecter ms_handle_connect 0x7f5f2813f650 2024-03-29T11:22:10.568 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.591+0000 7f5f24ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:10.568 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.591+0000 7f5f24ff9640 10 client.?.objecter ms_handle_connect 0x7f5f281457f0 2024-03-29T11:22:10.568 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.591+0000 7f5f24ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:10.568 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.591+0000 7f5f24ff9640 1 -- 172.21.15.17:0/698292028 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3240657034 0 0) 0x7f5f28142650 con 0x7f5f2813f650 2024-03-29T11:22:10.568 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.591+0000 7f5f24ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:22:10.568 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.591+0000 7f5f24ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:10.568 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.591+0000 7f5f24ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:10.568 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.591+0000 7f5f24ff9640 10 monclient(hunting): my global_id is 4344 2024-03-29T11:22:10.568 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.591+0000 7f5f24ff9640 1 -- 172.21.15.17:0/698292028 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f14004eb0 con 0x7f5f2813f650 2024-03-29T11:22:10.569 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.593+0000 7f5f24ff9640 1 -- 172.21.15.17:0/698292028 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3496683099 0 0) 0x7f5f2806eb00 con 0x7f5f281457f0 2024-03-29T11:22:10.569 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.593+0000 7f5f24ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:22:10.569 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.593+0000 7f5f24ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:10.569 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.593+0000 7f5f24ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:10.569 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.593+0000 7f5f24ff9640 10 monclient(hunting): my global_id is 4345 2024-03-29T11:22:10.569 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.593+0000 7f5f24ff9640 1 -- 172.21.15.17:0/698292028 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f28142650 con 0x7f5f281457f0 2024-03-29T11:22:10.571 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.594+0000 7f5f24ff9640 1 -- 172.21.15.17:0/698292028 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 889672477 0 0) 0x7f5f14004eb0 con 0x7f5f2813f650 2024-03-29T11:22:10.571 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.595+0000 7f5f24ff9640 1 -- 172.21.15.17:0/698292028 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f2806eb00 con 0x7f5f2813f650 2024-03-29T11:22:10.571 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.595+0000 7f5f24ff9640 1 -- 172.21.15.17:0/698292028 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f5f20001c60 con 0x7f5f2813f650 2024-03-29T11:22:10.571 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.595+0000 7f5f24ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:10.571 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.595+0000 7f5f24ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:10.571 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.595+0000 7f5f24ff9640 10 monclient(hunting): dump: 2024-03-29T11:22:10.571 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:22:10.572 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:22:10.572 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:10.572 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:10.572 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:10.572 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:10.572 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:22:10.572 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:22:10.572 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:22:10.572 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:10.572 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.595+0000 7f5f24ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:10.573 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.595+0000 7f5f24ff9640 1 -- 172.21.15.17:0/698292028 <== mon.0 v1:172.21.15.17:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1554694331 0 0) 0x7f5f2806eb00 con 0x7f5f2813f650 2024-03-29T11:22:10.573 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.595+0000 7f5f24ff9640 1 -- 172.21.15.17:0/698292028 >> v1:172.21.15.17:6790/0 conn(0x7f5f280a9d70 legacy=0x7f5f2813fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:10.573 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.595+0000 7f5f24ff9640 1 -- 172.21.15.17:0/698292028 >> v1:172.21.15.94:6789/0 conn(0x7f5f281457f0 legacy=0x7f5f281dbb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:10.573 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.595+0000 7f5f24ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:22:10.573 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.595+0000 7f5f24ff9640 1 monclient: found mon.a 2024-03-29T11:22:10.573 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.595+0000 7f5f24ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:10.574 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.595+0000 7f5f24ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:22:10.574 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.595+0000 7f5f24ff9640 1 -- 172.21.15.17:0/698292028 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f28142ae0 con 0x7f5f2813f650 2024-03-29T11:22:10.574 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.595+0000 7f5f24ff9640 10 monclient: _finish_auth 0 2024-03-29T11:22:10.574 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.595+0000 7f5f24ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:22:10.574 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.595+0000 7f5f24ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:10.574 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.595+0000 7f5f24ff9640 10 client.?.objecter ms_handle_connect 0x7f5f280a9d70 2024-03-29T11:22:10.574 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.595+0000 7f5f24ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:10.574 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.596+0000 7f5f24ff9640 1 -- 172.21.15.17:0/698292028 <== mon.0 v1:172.21.15.17:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f20004b60 con 0x7f5f2813f650 2024-03-29T11:22:10.574 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.596+0000 7f5f24ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:10.574 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.596+0000 7f5f25ffb640 4 set_mon_vals no callback set 2024-03-29T11:22:10.574 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.596+0000 7f5f24ff9640 1 -- 172.21.15.17:0/698292028 <== mon.0 v1:172.21.15.17:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f5f20004e70 con 0x7f5f2813f650 2024-03-29T11:22:10.574 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.596+0000 7f5f24ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:10.574 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.596+0000 7f5f24ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:10.574 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.596+0000 7f5f24ff9640 10 monclient: dump: 2024-03-29T11:22:10.574 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:22:10.575 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:22:10.575 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:10.575 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:10.575 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:10.575 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:10.575 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:22:10.575 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:22:10.575 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:22:10.575 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:10.575 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.597+0000 7f5f2e33e640 5 monclient: authenticate success, global_id 4344 2024-03-29T11:22:10.575 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.597+0000 7f5f2e33e640 10 monclient: _renew_subs 2024-03-29T11:22:10.575 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.597+0000 7f5f2e33e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:22:10.575 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.597+0000 7f5f2e33e640 1 -- 172.21.15.17:0/698292028 --> v1:172.21.15.17:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f281420a0 con 0x7f5f2813f650 2024-03-29T11:22:10.575 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.597+0000 7f5f2e33e640 10 client.4344.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:10.575 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.597+0000 7f5f2e33e640 10 monclient: _renew_subs 2024-03-29T11:22:10.576 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.597+0000 7f5f2e33e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:22:10.576 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.597+0000 7f5f2e33e640 1 -- 172.21.15.17:0/698292028 --> v1:172.21.15.17:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f28137930 con 0x7f5f2813f650 2024-03-29T11:22:10.576 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.597+0000 7f5f2e33e640 1 librados: init done 2024-03-29T11:22:10.576 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.599+0000 7f5f2e33e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:10.576 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.599+0000 7f5f2e33e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:10.576 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.599+0000 7f5f2e33e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:22:10.576 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.599+0000 7f5f2e33e640 1 -- 172.21.15.17:0/698292028 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f2806e920 con 0x7f5f2813f650 2024-03-29T11:22:10.581 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.603+0000 7f5f24ff9640 1 -- 172.21.15.17:0/698292028 <== mon.0 v1:172.21.15.17:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f5f2001ae40 con 0x7f5f2813f650 2024-03-29T11:22:10.581 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.604+0000 7f5f24ff9640 1 -- 172.21.15.17:0/698292028 <== mon.0 v1:172.21.15.17:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f5f20084de0 con 0x7f5f2813f650 2024-03-29T11:22:10.581 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.604+0000 7f5f24ff9640 10 client.4344.objecter ms_dispatch 0x7f5f28000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:10.581 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.604+0000 7f5f24ff9640 3 client.4344.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:10.581 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.604+0000 7f5f24ff9640 3 client.4344.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:10.581 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.604+0000 7f5f24ff9640 20 client.4344.objecter dump_active .. 0 homeless 2024-03-29T11:22:10.582 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.605+0000 7f5f27fff640 10 monclient: get_auth_request con 0x7f5f1406dde0 auth_method 0 2024-03-29T11:22:10.582 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.605+0000 7f5f24ff9640 10 client.4344.objecter ms_handle_connect 0x7f5f1406dde0 2024-03-29T11:22:10.587 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.611+0000 7f5f24ff9640 1 -- 172.21.15.17:0/698292028 <== mon.0 v1:172.21.15.17:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5f200b48b0 con 0x7f5f2813f650 2024-03-29T11:22:10.587 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.611+0000 7f5f24ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:10.587 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.611+0000 7f5f24ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:10.915 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.938+0000 7f5f2e33e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:10.915 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.938+0000 7f5f2e33e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:10.915 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.938+0000 7f5f2e33e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:22:10.915 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.938+0000 7f5f2e33e640 1 -- 172.21.15.17:0/698292028 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f28142800 con 0x7f5f2813f650 2024-03-29T11:22:10.916 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.939+0000 7f5f24ff9640 1 -- 172.21.15.17:0/698292028 <== mon.0 v1:172.21.15.17:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f5f28142800 con 0x7f5f2813f650 2024-03-29T11:22:10.916 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.939+0000 7f5f24ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:10.916 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.939+0000 7f5f24ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:10.916 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:22:10.917 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:10.919 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.943+0000 7f5f2e33e640 10 librados: watch_flush enter 2024-03-29T11:22:10.923 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.946+0000 7f5f2e33e640 10 librados: watch_flush exit 2024-03-29T11:22:10.923 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.946+0000 7f5f2e33e640 20 client.4344.objecter shutdown clearing up homeless session... 2024-03-29T11:22:10.923 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.946+0000 7f5f2e33e640 10 client.4344.objecter successfully canceled tick 2024-03-29T11:22:10.923 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.946+0000 7f5f2e33e640 1 -- 172.21.15.17:0/698292028 >> v1:172.21.15.17:6816/3236185052 conn(0x7f5f1406dde0 legacy=0x7f5f140702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:10.923 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.946+0000 7f5f2e33e640 10 monclient: shutdown 2024-03-29T11:22:10.923 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.946+0000 7f5f2e33e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:10.923 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.946+0000 7f5f2e33e640 1 -- 172.21.15.17:0/698292028 >> v1:172.21.15.17:6789/0 conn(0x7f5f2813f650 legacy=0x7f5f281db440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:10.923 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.946+0000 7f5f2e33e640 1 -- 172.21.15.17:0/698292028 shutdown_connections 2024-03-29T11:22:10.923 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.946+0000 7f5f2e33e640 1 -- 172.21.15.17:0/698292028 >> 172.21.15.17:0/698292028 conn(0x7f5f28066a00 msgr2=0x7f5f280b2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:10.923 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.946+0000 7f5f2e33e640 1 -- 172.21.15.17:0/698292028 shutdown_connections 2024-03-29T11:22:10.923 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.946+0000 7f5f2e33e640 1 -- 172.21.15.17:0/698292028 wait complete. 2024-03-29T11:22:10.924 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:10.946+0000 7f5f2e33e640 1 librados: shutdown 2024-03-29T11:22:10.943 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:17.749 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:17.914 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.938+0000 7f3bdca06640 10 monclient: get_monmap_and_config 2024-03-29T11:22:17.915 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.938+0000 7f3bdca06640 10 monclient: build_initial_monmap 2024-03-29T11:22:17.915 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.938+0000 7f3bdca06640 10 monclient: monmap: 2024-03-29T11:22:17.915 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:22:17.915 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:22:17.915 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:22:17.939059+0000 2024-03-29T11:22:17.915 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:22:17.939059+0000 2024-03-29T11:22:17.915 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:17.915 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:17.915 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:22:17.915 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:22:17.915 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:22:17.915 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:17.916 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.939+0000 7f3bdca06640 1 Processor -- start 2024-03-29T11:22:17.916 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.939+0000 7f3bdca06640 1 -- start start 2024-03-29T11:22:17.917 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.939+0000 7f3bdca06640 10 monclient: init 2024-03-29T11:22:17.917 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.940+0000 7f3bdca06640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:17.917 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.940+0000 7f3bdca06640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:22:17.917 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.940+0000 7f3bdca06640 10 monclient(hunting): picked mon.noname-c con 0x7f3bd80b0770 addr v1:172.21.15.94:6789/0 2024-03-29T11:22:17.917 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.940+0000 7f3bdca06640 10 monclient(hunting): picked mon.noname-b con 0x7f3bd814a700 addr v1:172.21.15.17:6790/0 2024-03-29T11:22:17.917 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.940+0000 7f3bdca06640 10 monclient(hunting): picked mon.noname-a con 0x7f3bd80ac5a0 addr v1:172.21.15.17:6789/0 2024-03-29T11:22:17.917 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.940+0000 7f3bdca06640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bd80aba70 con 0x7f3bd80ac5a0 2024-03-29T11:22:17.917 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.940+0000 7f3bdca06640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bd80aecb0 con 0x7f3bd80b0770 2024-03-29T11:22:17.917 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.940+0000 7f3bdca06640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bd80af150 con 0x7f3bd814a700 2024-03-29T11:22:17.917 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.940+0000 7f3bdca06640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:17.917 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.940+0000 7f3bdca06640 10 monclient(hunting): authenticate will time out at 1044.145406s 2024-03-29T11:22:17.917 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.940+0000 7f3bd6d9d640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f3bd80ac5a0 0x7f3bd8151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:56992/0 (socket says 172.21.15.17:56992) 2024-03-29T11:22:17.917 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.940+0000 7f3bd6d9d640 1 -- 172.21.15.17:0/1933589941 learned_addr learned my addr 172.21.15.17:0/1933589941 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:22:17.920 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.941+0000 7f3bd559a640 1 -- 172.21.15.17:0/1933589941 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1928424318 0 0) 0x7f3bd80aecb0 con 0x7f3bd80b0770 2024-03-29T11:22:17.920 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.941+0000 7f3bd559a640 10 monclient(hunting): _negotiate 2024-03-29T11:22:17.920 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.941+0000 7f3bd559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:17.920 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.941+0000 7f3bd559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:17.920 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.941+0000 7f3bd559a640 10 monclient(hunting): my global_id is 4348 2024-03-29T11:22:17.921 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.941+0000 7f3bd559a640 1 -- 172.21.15.17:0/1933589941 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bc4003520 con 0x7f3bd80b0770 2024-03-29T11:22:17.921 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.941+0000 7f3bd559a640 1 -- 172.21.15.17:0/1933589941 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3756851084 0 0) 0x7f3bd80aba70 con 0x7f3bd80ac5a0 2024-03-29T11:22:17.921 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.941+0000 7f3bd559a640 10 monclient(hunting): _negotiate 2024-03-29T11:22:17.921 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.941+0000 7f3bd559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:17.921 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.941+0000 7f3bd559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:17.921 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.941+0000 7f3bd559a640 10 monclient(hunting): my global_id is 4347 2024-03-29T11:22:17.921 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.941+0000 7f3bd559a640 1 -- 172.21.15.17:0/1933589941 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bd80aecb0 con 0x7f3bd80ac5a0 2024-03-29T11:22:17.921 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.941+0000 7f3bd559a640 1 -- 172.21.15.17:0/1933589941 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1789737793 0 0) 0x7f3bd80af150 con 0x7f3bd814a700 2024-03-29T11:22:17.921 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.941+0000 7f3bd559a640 10 monclient(hunting): _negotiate 2024-03-29T11:22:17.921 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.941+0000 7f3bd559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:17.921 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.941+0000 7f3bd559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:17.921 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.942+0000 7f3bd559a640 10 monclient(hunting): my global_id is 4370 2024-03-29T11:22:17.921 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.942+0000 7f3bd559a640 1 -- 172.21.15.17:0/1933589941 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bd80aba70 con 0x7f3bd814a700 2024-03-29T11:22:17.921 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.942+0000 7f3bd559a640 1 -- 172.21.15.17:0/1933589941 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1138117418 0 0) 0x7f3bc4003520 con 0x7f3bd80b0770 2024-03-29T11:22:17.921 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.942+0000 7f3bd559a640 1 -- 172.21.15.17:0/1933589941 >> v1:172.21.15.17:6790/0 conn(0x7f3bd814a700 legacy=0x7f3bd80abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:17.922 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.942+0000 7f3bd559a640 1 -- 172.21.15.17:0/1933589941 >> v1:172.21.15.17:6789/0 conn(0x7f3bd80ac5a0 legacy=0x7f3bd8151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:17.922 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.942+0000 7f3bd559a640 10 monclient: _finish_hunting 0 2024-03-29T11:22:17.922 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.942+0000 7f3bd559a640 1 monclient: found mon.noname-c 2024-03-29T11:22:17.922 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.942+0000 7f3bd559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:17.922 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.942+0000 7f3bd559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6789/0 2024-03-29T11:22:17.922 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.942+0000 7f3bd559a640 1 -- 172.21.15.17:0/1933589941 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bd80ac9d0 con 0x7f3bd80b0770 2024-03-29T11:22:17.922 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.942+0000 7f3bd559a640 10 monclient: _finish_auth 0 2024-03-29T11:22:17.922 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.942+0000 7f3bd559a640 10 monclient: _check_auth_tickets 2024-03-29T11:22:17.922 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.942+0000 7f3bd559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:17.922 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.942+0000 7f3bdca06640 5 monclient: authenticate success, global_id 4348 2024-03-29T11:22:17.922 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.942+0000 7f3bdca06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:17.922 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.942+0000 7f3bd559a640 1 -- 172.21.15.17:0/1933589941 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f3bc00016b0 con 0x7f3bd80b0770 2024-03-29T11:22:17.922 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.942+0000 7f3bd559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:17.922 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.942+0000 7f3bd559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:17.923 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.942+0000 7f3bd559a640 10 monclient: dump: 2024-03-29T11:22:17.923 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:22:17.923 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:22:17.923 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:17.923 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:17.923 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:17.923 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:17.923 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:22:17.923 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:22:17.923 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:22:17.923 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:17.923 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.942+0000 7f3bdca06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:17.923 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.942+0000 7f3bd559a640 1 -- 172.21.15.17:0/1933589941 <== mon.1 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3bc00018d0 con 0x7f3bd80b0770 2024-03-29T11:22:17.923 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.942+0000 7f3bd559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:17.923 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.942+0000 7f3bdca06640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:17.923 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.942+0000 7f3bdca06640 4 set_mon_vals no callback set 2024-03-29T11:22:17.924 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.943+0000 7f3bd559a640 1 -- 172.21.15.17:0/1933589941 <== mon.1 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f3bc0001c60 con 0x7f3bd80b0770 2024-03-29T11:22:17.924 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.943+0000 7f3bd559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:17.924 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.943+0000 7f3bd559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:17.924 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.943+0000 7f3bd559a640 10 monclient: dump: 2024-03-29T11:22:17.924 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:22:17.924 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:22:17.924 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:17.924 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:17.924 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:17.924 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:17.924 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:22:17.924 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:22:17.924 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:22:17.924 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:17.924 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.943+0000 7f3bdca06640 10 monclient: shutdown 2024-03-29T11:22:17.924 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.943+0000 7f3bdca06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:17.924 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.943+0000 7f3bdca06640 1 -- 172.21.15.17:0/1933589941 >> v1:172.21.15.94:6789/0 conn(0x7f3bd80b0770 legacy=0x7f3bd8149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:17.925 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.944+0000 7f3bdca06640 1 -- 172.21.15.17:0/1933589941 shutdown_connections 2024-03-29T11:22:17.925 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.944+0000 7f3bdca06640 1 -- 172.21.15.17:0/1933589941 >> 172.21.15.17:0/1933589941 conn(0x7f3bd806b660 msgr2=0x7f3bd806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:17.925 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.944+0000 7f3bdca06640 1 -- 172.21.15.17:0/1933589941 shutdown_connections 2024-03-29T11:22:17.925 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.944+0000 7f3bdca06640 1 -- 172.21.15.17:0/1933589941 wait complete. 2024-03-29T11:22:17.928 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.945+0000 7f3bdca06640 10 monclient: build_initial_monmap 2024-03-29T11:22:17.928 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.945+0000 7f3bdca06640 10 monclient: monmap: 2024-03-29T11:22:17.928 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:22:17.928 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:17.928 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:22:17.928 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:22:17.928 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:17.928 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:17.928 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:22:17.928 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:22:17.928 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:22:17.928 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:17.928 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.945+0000 7f3bdca06640 1 librados: starting msgr at 2024-03-29T11:22:17.928 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.945+0000 7f3bdca06640 1 librados: starting objecter 2024-03-29T11:22:17.929 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.945+0000 7f3bdca06640 1 Processor -- start 2024-03-29T11:22:17.929 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.946+0000 7f3bdca06640 1 -- start start 2024-03-29T11:22:17.929 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.946+0000 7f3bdca06640 1 librados: setting wanted keys 2024-03-29T11:22:17.929 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.946+0000 7f3bdca06640 1 librados: calling monclient init 2024-03-29T11:22:17.929 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.946+0000 7f3bdca06640 10 monclient: init 2024-03-29T11:22:17.929 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.946+0000 7f3bdca06640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:17.929 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.946+0000 7f3bdca06640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:17.929 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.946+0000 7f3bdca06640 10 monclient(hunting): picked mon.noname-c con 0x7f3bd80ac5a0 addr v1:172.21.15.17:6790/0 2024-03-29T11:22:17.929 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.946+0000 7f3bdca06640 10 monclient(hunting): picked mon.noname-b con 0x7f3bd80b0770 addr v1:172.21.15.94:6789/0 2024-03-29T11:22:17.929 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.946+0000 7f3bdca06640 10 monclient(hunting): picked mon.noname-a con 0x7f3bd814a700 addr v1:172.21.15.17:6789/0 2024-03-29T11:22:17.929 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.946+0000 7f3bdca06640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bd8128650 con 0x7f3bd814a700 2024-03-29T11:22:17.929 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.946+0000 7f3bdca06640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bd81262d0 con 0x7f3bd80b0770 2024-03-29T11:22:17.929 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.946+0000 7f3bdca06640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bd81268f0 con 0x7f3bd80ac5a0 2024-03-29T11:22:17.929 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.946+0000 7f3bdca06640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:17.930 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.946+0000 7f3bdca06640 10 monclient(hunting): authenticate will time out at 1044.151729s 2024-03-29T11:22:17.930 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.947+0000 7f3bd5d9b640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f3bd80b0770 0x7f3bd8145240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:55630/0 (socket says 172.21.15.17:55630) 2024-03-29T11:22:17.930 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.947+0000 7f3bd5d9b640 1 -- 172.21.15.17:0/205222828 learned_addr learned my addr 172.21.15.17:0/205222828 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:22:17.930 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.947+0000 7f3bceffd640 10 client.?.objecter ms_handle_connect 0x7f3bd80b0770 2024-03-29T11:22:17.930 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.947+0000 7f3bceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:17.930 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.947+0000 7f3bceffd640 10 client.?.objecter ms_handle_connect 0x7f3bd814a700 2024-03-29T11:22:17.930 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.947+0000 7f3bceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:17.930 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.947+0000 7f3bceffd640 1 -- 172.21.15.17:0/205222828 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3785955317 0 0) 0x7f3bd81262d0 con 0x7f3bd80b0770 2024-03-29T11:22:17.930 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.947+0000 7f3bceffd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:17.930 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.947+0000 7f3bceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:17.930 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.947+0000 7f3bceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:17.930 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.947+0000 7f3bceffd640 10 monclient(hunting): my global_id is 4351 2024-03-29T11:22:17.930 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.947+0000 7f3bceffd640 1 -- 172.21.15.17:0/205222828 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bb8004eb0 con 0x7f3bd80b0770 2024-03-29T11:22:17.930 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.947+0000 7f3bceffd640 1 -- 172.21.15.17:0/205222828 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 953842023 0 0) 0x7f3bd8128650 con 0x7f3bd814a700 2024-03-29T11:22:17.931 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.947+0000 7f3bceffd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:17.931 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.947+0000 7f3bceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:17.931 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.947+0000 7f3bceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:17.931 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.947+0000 7f3bceffd640 10 monclient(hunting): my global_id is 4350 2024-03-29T11:22:17.931 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.947+0000 7f3bceffd640 1 -- 172.21.15.17:0/205222828 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bd81262d0 con 0x7f3bd814a700 2024-03-29T11:22:17.931 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.948+0000 7f3bceffd640 1 -- 172.21.15.17:0/205222828 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3910368924 0 0) 0x7f3bb8004eb0 con 0x7f3bd80b0770 2024-03-29T11:22:17.931 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.948+0000 7f3bceffd640 1 -- 172.21.15.17:0/205222828 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3bd8128650 con 0x7f3bd80b0770 2024-03-29T11:22:17.931 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.948+0000 7f3bceffd640 1 -- 172.21.15.17:0/205222828 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f3bc8001990 con 0x7f3bd80b0770 2024-03-29T11:22:17.931 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.948+0000 7f3bceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:17.931 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.948+0000 7f3bceffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:17.931 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.948+0000 7f3bceffd640 10 monclient(hunting): dump: 2024-03-29T11:22:17.931 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:22:17.931 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:22:17.931 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:17.931 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:17.932 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:17.932 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:17.932 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:22:17.932 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:22:17.932 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:22:17.932 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:17.932 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.948+0000 7f3bceffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:17.932 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.948+0000 7f3bceffd640 10 client.?.objecter ms_handle_connect 0x7f3bd80ac5a0 2024-03-29T11:22:17.932 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.948+0000 7f3bceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:17.932 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.948+0000 7f3bceffd640 1 -- 172.21.15.17:0/205222828 <== mon.1 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2001510562 0 0) 0x7f3bd8128650 con 0x7f3bd80b0770 2024-03-29T11:22:17.932 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.948+0000 7f3bceffd640 1 -- 172.21.15.17:0/205222828 >> v1:172.21.15.17:6790/0 conn(0x7f3bd80ac5a0 legacy=0x7f3bd8126e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:17.932 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.948+0000 7f3bceffd640 1 -- 172.21.15.17:0/205222828 >> v1:172.21.15.17:6789/0 conn(0x7f3bd814a700 legacy=0x7f3bd81e6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:17.932 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.948+0000 7f3bceffd640 10 monclient: _finish_hunting 0 2024-03-29T11:22:17.932 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.948+0000 7f3bceffd640 1 monclient: found mon.b 2024-03-29T11:22:17.932 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.948+0000 7f3bceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:17.932 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.948+0000 7f3bceffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:22:17.933 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.948+0000 7f3bceffd640 1 -- 172.21.15.17:0/205222828 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bd81239d0 con 0x7f3bd80b0770 2024-03-29T11:22:17.933 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.948+0000 7f3bceffd640 10 monclient: _finish_auth 0 2024-03-29T11:22:17.933 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.948+0000 7f3bceffd640 10 monclient: _check_auth_tickets 2024-03-29T11:22:17.933 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.948+0000 7f3bceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:17.933 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.948+0000 7f3bdca06640 5 monclient: authenticate success, global_id 4351 2024-03-29T11:22:17.933 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.948+0000 7f3bdca06640 10 monclient: _renew_subs 2024-03-29T11:22:17.933 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.948+0000 7f3bdca06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:22:17.933 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.948+0000 7f3bdca06640 1 -- 172.21.15.17:0/205222828 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bd8126580 con 0x7f3bd80b0770 2024-03-29T11:22:17.933 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.948+0000 7f3bdca06640 10 client.4351.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:17.933 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.948+0000 7f3bdca06640 10 monclient: _renew_subs 2024-03-29T11:22:17.933 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.948+0000 7f3bdca06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:22:17.933 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.948+0000 7f3bdca06640 1 -- 172.21.15.17:0/205222828 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bd81dfab0 con 0x7f3bd80b0770 2024-03-29T11:22:17.933 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.948+0000 7f3bdca06640 1 librados: init done 2024-03-29T11:22:17.933 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.948+0000 7f3bceffd640 1 -- 172.21.15.17:0/205222828 <== mon.1 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3bc80035f0 con 0x7f3bd80b0770 2024-03-29T11:22:17.933 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.948+0000 7f3bceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:17.933 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.948+0000 7f3bceffd640 1 -- 172.21.15.17:0/205222828 <== mon.1 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f3bc80039c0 con 0x7f3bd80b0770 2024-03-29T11:22:17.933 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.948+0000 7f3bceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:17.934 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.948+0000 7f3bceffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:17.934 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.948+0000 7f3bceffd640 10 monclient: dump: 2024-03-29T11:22:17.934 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:22:17.934 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:22:17.934 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:17.934 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:17.934 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:17.934 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:17.934 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:22:17.934 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:22:17.934 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:22:17.934 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:17.934 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.948+0000 7f3bd4d99640 4 set_mon_vals no callback set 2024-03-29T11:22:17.934 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.950+0000 7f3bdca06640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:17.934 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.950+0000 7f3bdca06640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:17.935 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.951+0000 7f3bdca06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:22:17.935 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.951+0000 7f3bdca06640 1 -- 172.21.15.17:0/205222828 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bbc005080 con 0x7f3bd80b0770 2024-03-29T11:22:17.935 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.951+0000 7f3bceffd640 1 -- 172.21.15.17:0/205222828 <== mon.1 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f3bc8003b20 con 0x7f3bd80b0770 2024-03-29T11:22:17.935 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.952+0000 7f3bceffd640 1 -- 172.21.15.17:0/205222828 <== mon.1 v1:172.21.15.94:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f3bc8084a30 con 0x7f3bd80b0770 2024-03-29T11:22:17.935 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.952+0000 7f3bceffd640 10 client.4351.objecter ms_dispatch 0x7f3bd8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:17.935 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.952+0000 7f3bceffd640 3 client.4351.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:17.935 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.952+0000 7f3bceffd640 3 client.4351.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:17.935 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.952+0000 7f3bceffd640 20 client.4351.objecter dump_active .. 0 homeless 2024-03-29T11:22:17.936 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.952+0000 7f3bd659c640 10 monclient: get_auth_request con 0x7f3bb806db60 auth_method 0 2024-03-29T11:22:17.936 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.952+0000 7f3bceffd640 10 client.4351.objecter ms_handle_connect 0x7f3bb806db60 2024-03-29T11:22:17.936 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.958+0000 7f3bceffd640 1 -- 172.21.15.17:0/205222828 <== mon.1 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3bc8050310 con 0x7f3bd80b0770 2024-03-29T11:22:17.936 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.958+0000 7f3bceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:17.936 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:17.958+0000 7f3bceffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:18.335 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:18.358+0000 7f3bdca06640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:18.335 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:18.359+0000 7f3bdca06640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:18.335 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:18.359+0000 7f3bdca06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:22:18.336 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:18.359+0000 7f3bdca06640 1 -- 172.21.15.17:0/205222828 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3bbc005330 con 0x7f3bd80b0770 2024-03-29T11:22:18.337 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:18.360+0000 7f3bceffd640 1 -- 172.21.15.17:0/205222828 <== mon.1 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f3bc8052240 con 0x7f3bd80b0770 2024-03-29T11:22:18.337 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:18.360+0000 7f3bceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:18.337 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:18.360+0000 7f3bceffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:18.337 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:22:18.337 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:18.340 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:18.363+0000 7f3bdca06640 10 librados: watch_flush enter 2024-03-29T11:22:18.340 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:18.363+0000 7f3bdca06640 10 librados: watch_flush exit 2024-03-29T11:22:18.340 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:18.363+0000 7f3bdca06640 20 client.4351.objecter shutdown clearing up homeless session... 2024-03-29T11:22:18.340 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:18.363+0000 7f3bdca06640 10 client.4351.objecter successfully canceled tick 2024-03-29T11:22:18.340 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:18.364+0000 7f3bdca06640 1 -- 172.21.15.17:0/205222828 >> v1:172.21.15.17:6816/3236185052 conn(0x7f3bb806db60 legacy=0x7f3bb8070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:18.340 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:18.364+0000 7f3bdca06640 10 monclient: shutdown 2024-03-29T11:22:18.340 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:18.364+0000 7f3bdca06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:18.340 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:18.364+0000 7f3bdca06640 1 -- 172.21.15.17:0/205222828 >> v1:172.21.15.94:6789/0 conn(0x7f3bd80b0770 legacy=0x7f3bd8145240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:18.341 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:18.364+0000 7f3bdca06640 1 -- 172.21.15.17:0/205222828 shutdown_connections 2024-03-29T11:22:18.341 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:18.364+0000 7f3bdca06640 1 -- 172.21.15.17:0/205222828 >> 172.21.15.17:0/205222828 conn(0x7f3bd806b660 msgr2=0x7f3bd80b3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:18.341 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:18.364+0000 7f3bdca06640 1 -- 172.21.15.17:0/205222828 shutdown_connections 2024-03-29T11:22:18.341 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:18.365+0000 7f3bdca06640 1 -- 172.21.15.17:0/205222828 wait complete. 2024-03-29T11:22:18.341 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:18.365+0000 7f3bdca06640 1 librados: shutdown 2024-03-29T11:22:18.357 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:25.163 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:25.312 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.335+0000 7fc1b53e9640 10 monclient: get_monmap_and_config 2024-03-29T11:22:25.313 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.335+0000 7fc1b53e9640 10 monclient: build_initial_monmap 2024-03-29T11:22:25.313 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.335+0000 7fc1b53e9640 10 monclient: monmap: 2024-03-29T11:22:25.313 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:22:25.313 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:22:25.313 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:22:25.336870+0000 2024-03-29T11:22:25.313 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:22:25.336870+0000 2024-03-29T11:22:25.313 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:25.313 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:25.313 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:22:25.313 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:22:25.313 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:22:25.313 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:25.313 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.336+0000 7fc1b53e9640 1 Processor -- start 2024-03-29T11:22:25.314 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.336+0000 7fc1b53e9640 1 -- start start 2024-03-29T11:22:25.314 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.336+0000 7fc1b53e9640 10 monclient: init 2024-03-29T11:22:25.327 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.337+0000 7fc1b53e9640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:25.327 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.337+0000 7fc1b53e9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:25.327 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.337+0000 7fc1b53e9640 10 monclient(hunting): picked mon.noname-b con 0x7fc1b01422b0 addr v1:172.21.15.17:6790/0 2024-03-29T11:22:25.327 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.337+0000 7fc1b53e9640 10 monclient(hunting): picked mon.noname-c con 0x7fc1b0142690 addr v1:172.21.15.94:6789/0 2024-03-29T11:22:25.327 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.337+0000 7fc1b53e9640 10 monclient(hunting): picked mon.noname-a con 0x7fc1b00b17d0 addr v1:172.21.15.17:6789/0 2024-03-29T11:22:25.327 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.337+0000 7fc1b53e9640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1b0142c80 con 0x7fc1b00b17d0 2024-03-29T11:22:25.327 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.337+0000 7fc1b53e9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1b00aae40 con 0x7fc1b0142690 2024-03-29T11:22:25.327 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.337+0000 7fc1b53e9640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1b013fbd0 con 0x7fc1b01422b0 2024-03-29T11:22:25.327 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.337+0000 7fc1b53e9640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:25.328 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.337+0000 7fc1b53e9640 10 monclient(hunting): authenticate will time out at 1051.542863s 2024-03-29T11:22:25.328 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.338+0000 7fc1aeffd640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7fc1b01422b0 0x7fc1b013f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:57660/0 (socket says 172.21.15.17:57660) 2024-03-29T11:22:25.328 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.338+0000 7fc1aeffd640 1 -- 172.21.15.17:0/2420738074 learned_addr learned my addr 172.21.15.17:0/2420738074 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:22:25.328 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.338+0000 7fc1adffb640 1 -- 172.21.15.17:0/2420738074 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1161131279 0 0) 0x7fc1b013fbd0 con 0x7fc1b01422b0 2024-03-29T11:22:25.328 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.338+0000 7fc1adffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:25.328 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.338+0000 7fc1adffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:25.328 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.338+0000 7fc1adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:25.328 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.338+0000 7fc1adffb640 10 monclient(hunting): my global_id is 4376 2024-03-29T11:22:25.328 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.338+0000 7fc1adffb640 1 -- 172.21.15.17:0/2420738074 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc19c003520 con 0x7fc1b01422b0 2024-03-29T11:22:25.328 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.338+0000 7fc1adffb640 1 -- 172.21.15.17:0/2420738074 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 713904223 0 0) 0x7fc1b00aae40 con 0x7fc1b0142690 2024-03-29T11:22:25.328 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.338+0000 7fc1adffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:25.328 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.338+0000 7fc1adffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:25.328 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.338+0000 7fc1adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:25.328 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.338+0000 7fc1adffb640 10 monclient(hunting): my global_id is 4354 2024-03-29T11:22:25.328 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.338+0000 7fc1adffb640 1 -- 172.21.15.17:0/2420738074 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1b013fbd0 con 0x7fc1b0142690 2024-03-29T11:22:25.329 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.339+0000 7fc1adffb640 1 -- 172.21.15.17:0/2420738074 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2647525003 0 0) 0x7fc1b0142c80 con 0x7fc1b00b17d0 2024-03-29T11:22:25.329 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.339+0000 7fc1adffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:25.329 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.339+0000 7fc1adffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:25.329 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.339+0000 7fc1adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:25.329 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.339+0000 7fc1adffb640 10 monclient(hunting): my global_id is 4353 2024-03-29T11:22:25.329 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.339+0000 7fc1adffb640 1 -- 172.21.15.17:0/2420738074 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1b00aae40 con 0x7fc1b00b17d0 2024-03-29T11:22:25.329 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.339+0000 7fc1adffb640 1 -- 172.21.15.17:0/2420738074 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1599698004 0 0) 0x7fc1b013fbd0 con 0x7fc1b0142690 2024-03-29T11:22:25.329 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.339+0000 7fc1adffb640 1 -- 172.21.15.17:0/2420738074 >> v1:172.21.15.17:6790/0 conn(0x7fc1b01422b0 legacy=0x7fc1b013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:25.329 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.339+0000 7fc1adffb640 1 -- 172.21.15.17:0/2420738074 >> v1:172.21.15.17:6789/0 conn(0x7fc1b00b17d0 legacy=0x7fc1b00b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:25.329 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.339+0000 7fc1adffb640 10 monclient: _finish_hunting 0 2024-03-29T11:22:25.329 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.339+0000 7fc1adffb640 1 monclient: found mon.noname-c 2024-03-29T11:22:25.329 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.339+0000 7fc1adffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:25.329 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.339+0000 7fc1adffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6789/0 2024-03-29T11:22:25.329 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.339+0000 7fc1adffb640 1 -- 172.21.15.17:0/2420738074 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b0151940 con 0x7fc1b0142690 2024-03-29T11:22:25.329 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.339+0000 7fc1adffb640 10 monclient: _finish_auth 0 2024-03-29T11:22:25.330 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.339+0000 7fc1adffb640 10 monclient: _check_auth_tickets 2024-03-29T11:22:25.330 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.339+0000 7fc1adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:25.330 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.339+0000 7fc1adffb640 1 -- 172.21.15.17:0/2420738074 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fc1a0001670 con 0x7fc1b0142690 2024-03-29T11:22:25.330 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.339+0000 7fc1adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:25.330 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.339+0000 7fc1adffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:25.330 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.339+0000 7fc1adffb640 10 monclient: dump: 2024-03-29T11:22:25.330 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:22:25.330 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:22:25.331 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:25.331 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:25.331 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:25.331 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:25.331 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:22:25.331 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:22:25.332 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:22:25.332 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:25.332 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.339+0000 7fc1b53e9640 5 monclient: authenticate success, global_id 4354 2024-03-29T11:22:25.332 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.339+0000 7fc1b53e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:25.332 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.339+0000 7fc1adffb640 1 -- 172.21.15.17:0/2420738074 <== mon.1 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc1a0001890 con 0x7fc1b0142690 2024-03-29T11:22:25.332 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.339+0000 7fc1adffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:25.332 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.339+0000 7fc1b53e9640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:25.332 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.339+0000 7fc1b53e9640 4 set_mon_vals no callback set 2024-03-29T11:22:25.332 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.339+0000 7fc1adffb640 1 -- 172.21.15.17:0/2420738074 <== mon.1 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fc1a0001ba0 con 0x7fc1b0142690 2024-03-29T11:22:25.332 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.339+0000 7fc1adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:25.333 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.339+0000 7fc1adffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:25.333 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.339+0000 7fc1adffb640 10 monclient: dump: 2024-03-29T11:22:25.333 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:22:25.333 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:22:25.333 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:25.333 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:25.333 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:25.333 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:25.333 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:22:25.333 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:22:25.333 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:22:25.333 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:25.333 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.340+0000 7fc1b53e9640 10 monclient: shutdown 2024-03-29T11:22:25.333 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.340+0000 7fc1b53e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:25.334 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.340+0000 7fc1b53e9640 1 -- 172.21.15.17:0/2420738074 >> v1:172.21.15.94:6789/0 conn(0x7fc1b0142690 legacy=0x7fc1b00b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:25.334 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.340+0000 7fc1b53e9640 1 -- 172.21.15.17:0/2420738074 shutdown_connections 2024-03-29T11:22:25.334 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.340+0000 7fc1b53e9640 1 -- 172.21.15.17:0/2420738074 >> 172.21.15.17:0/2420738074 conn(0x7fc1b0066a00 msgr2=0x7fc1b0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:25.334 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.340+0000 7fc1b53e9640 1 -- 172.21.15.17:0/2420738074 shutdown_connections 2024-03-29T11:22:25.334 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.340+0000 7fc1b53e9640 1 -- 172.21.15.17:0/2420738074 wait complete. 2024-03-29T11:22:25.334 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.341+0000 7fc1b53e9640 10 monclient: build_initial_monmap 2024-03-29T11:22:25.334 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.341+0000 7fc1b53e9640 10 monclient: monmap: 2024-03-29T11:22:25.334 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:22:25.334 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:25.334 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:22:25.335 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:22:25.335 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:25.335 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:25.335 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:22:25.335 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:22:25.335 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:22:25.335 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:25.335 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.341+0000 7fc1b53e9640 1 librados: starting msgr at 2024-03-29T11:22:25.335 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.341+0000 7fc1b53e9640 1 librados: starting objecter 2024-03-29T11:22:25.335 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.342+0000 7fc1b53e9640 1 Processor -- start 2024-03-29T11:22:25.335 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.342+0000 7fc1b53e9640 1 -- start start 2024-03-29T11:22:25.335 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.342+0000 7fc1b53e9640 1 librados: setting wanted keys 2024-03-29T11:22:25.335 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.342+0000 7fc1b53e9640 1 librados: calling monclient init 2024-03-29T11:22:25.336 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.342+0000 7fc1b53e9640 10 monclient: init 2024-03-29T11:22:25.336 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.342+0000 7fc1b53e9640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:25.336 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.342+0000 7fc1b53e9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:22:25.336 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.342+0000 7fc1b53e9640 10 monclient(hunting): picked mon.noname-a con 0x7fc1b00b17d0 addr v1:172.21.15.17:6789/0 2024-03-29T11:22:25.336 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.342+0000 7fc1b53e9640 10 monclient(hunting): picked mon.noname-b con 0x7fc1b01422b0 addr v1:172.21.15.94:6789/0 2024-03-29T11:22:25.336 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.342+0000 7fc1b53e9640 10 monclient(hunting): picked mon.noname-c con 0x7fc1b0142690 addr v1:172.21.15.17:6790/0 2024-03-29T11:22:25.336 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.342+0000 7fc1b53e9640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1b006e050 con 0x7fc1b00b17d0 2024-03-29T11:22:25.337 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.342+0000 7fc1b53e9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1b006e330 con 0x7fc1b01422b0 2024-03-29T11:22:25.337 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.342+0000 7fc1b53e9640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1b0131a40 con 0x7fc1b0142690 2024-03-29T11:22:25.337 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.342+0000 7fc1b53e9640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:25.337 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.342+0000 7fc1b53e9640 10 monclient(hunting): authenticate will time out at 1051.547841s 2024-03-29T11:22:25.337 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.343+0000 7fc1af7fe640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7fc1b0142690 0x7fc1b01e6190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:57980/0 (socket says 172.21.15.17:57980) 2024-03-29T11:22:25.337 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.343+0000 7fc1af7fe640 1 -- 172.21.15.17:0/3814102075 learned_addr learned my addr 172.21.15.17:0/3814102075 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:22:25.337 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.343+0000 7fc1a77fe640 10 client.?.objecter ms_handle_connect 0x7fc1b0142690 2024-03-29T11:22:25.337 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.343+0000 7fc1a77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:25.338 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.343+0000 7fc1a77fe640 10 client.?.objecter ms_handle_connect 0x7fc1b01422b0 2024-03-29T11:22:25.338 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.343+0000 7fc1a77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:25.338 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.343+0000 7fc1a77fe640 1 -- 172.21.15.17:0/3814102075 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2604604857 0 0) 0x7fc1b0131a40 con 0x7fc1b0142690 2024-03-29T11:22:25.338 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.343+0000 7fc1a77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:25.338 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.343+0000 7fc1a77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:25.338 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.343+0000 7fc1a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:25.338 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.343+0000 7fc1a77fe640 10 monclient(hunting): my global_id is 4379 2024-03-29T11:22:25.338 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.343+0000 7fc1a77fe640 1 -- 172.21.15.17:0/3814102075 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc190004eb0 con 0x7fc1b0142690 2024-03-29T11:22:25.338 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.343+0000 7fc1a77fe640 10 client.?.objecter ms_handle_connect 0x7fc1b00b17d0 2024-03-29T11:22:25.338 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.343+0000 7fc1a77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:25.338 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.343+0000 7fc1a77fe640 1 -- 172.21.15.17:0/3814102075 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2686743142 0 0) 0x7fc1b006e330 con 0x7fc1b01422b0 2024-03-29T11:22:25.338 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.344+0000 7fc1a77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:25.338 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.344+0000 7fc1a77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:25.338 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.344+0000 7fc1a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:25.339 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.344+0000 7fc1a77fe640 10 monclient(hunting): my global_id is 4357 2024-03-29T11:22:25.339 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.344+0000 7fc1a77fe640 1 -- 172.21.15.17:0/3814102075 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1b0131a40 con 0x7fc1b01422b0 2024-03-29T11:22:25.339 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.344+0000 7fc1a77fe640 1 -- 172.21.15.17:0/3814102075 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1948594184 0 0) 0x7fc1b006e050 con 0x7fc1b00b17d0 2024-03-29T11:22:25.339 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.344+0000 7fc1a77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:25.339 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.344+0000 7fc1a77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:25.339 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.344+0000 7fc1a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:25.339 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.344+0000 7fc1a77fe640 10 monclient(hunting): my global_id is 4356 2024-03-29T11:22:25.339 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.344+0000 7fc1a77fe640 1 -- 172.21.15.17:0/3814102075 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1b006e330 con 0x7fc1b00b17d0 2024-03-29T11:22:25.339 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.344+0000 7fc1a77fe640 1 -- 172.21.15.17:0/3814102075 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 319930359 0 0) 0x7fc190004eb0 con 0x7fc1b0142690 2024-03-29T11:22:25.340 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.344+0000 7fc1a77fe640 1 -- 172.21.15.17:0/3814102075 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1b006e050 con 0x7fc1b0142690 2024-03-29T11:22:25.340 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.344+0000 7fc1a77fe640 1 -- 172.21.15.17:0/3814102075 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fc1a8001d10 con 0x7fc1b0142690 2024-03-29T11:22:25.340 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.344+0000 7fc1a77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:25.340 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.344+0000 7fc1a77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:25.340 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.344+0000 7fc1a77fe640 10 monclient(hunting): dump: 2024-03-29T11:22:25.340 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:22:25.341 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:22:25.341 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:25.341 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:25.341 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:25.341 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:25.341 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:22:25.342 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:22:25.342 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:22:25.342 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:25.342 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.344+0000 7fc1a77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:25.342 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.344+0000 7fc1a77fe640 1 -- 172.21.15.17:0/3814102075 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1066896703 0 0) 0x7fc1b0131a40 con 0x7fc1b01422b0 2024-03-29T11:22:25.342 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.344+0000 7fc1a77fe640 1 -- 172.21.15.17:0/3814102075 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc190004eb0 con 0x7fc1b01422b0 2024-03-29T11:22:25.342 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.344+0000 7fc1a77fe640 1 -- 172.21.15.17:0/3814102075 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fc1a0001530 con 0x7fc1b01422b0 2024-03-29T11:22:25.342 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.344+0000 7fc1a77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:25.342 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.344+0000 7fc1a77fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:25.342 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.344+0000 7fc1a77fe640 10 monclient(hunting): dump: 2024-03-29T11:22:25.342 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:22:25.342 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:22:25.342 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:25.343 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:25.343 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:25.343 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:25.343 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:22:25.343 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:22:25.343 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:22:25.343 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:25.343 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.344+0000 7fc1a77fe640 1 -- 172.21.15.17:0/3814102075 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2304945858 0 0) 0x7fc1b006e330 con 0x7fc1b00b17d0 2024-03-29T11:22:25.343 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.344+0000 7fc1a77fe640 1 -- 172.21.15.17:0/3814102075 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1b0131a40 con 0x7fc1b00b17d0 2024-03-29T11:22:25.343 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.345+0000 7fc1a77fe640 1 -- 172.21.15.17:0/3814102075 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fc198001230 con 0x7fc1b00b17d0 2024-03-29T11:22:25.343 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.345+0000 7fc1a77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:25.343 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.345+0000 7fc1a77fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:25.343 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.345+0000 7fc1a77fe640 10 monclient(hunting): dump: 2024-03-29T11:22:25.343 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:22:25.344 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:22:25.344 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:25.344 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:25.344 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:25.344 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:25.344 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:22:25.344 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:22:25.344 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:22:25.344 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:25.344 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.345+0000 7fc1a77fe640 1 -- 172.21.15.17:0/3814102075 <== mon.2 v1:172.21.15.17:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4261959975 0 0) 0x7fc1b006e050 con 0x7fc1b0142690 2024-03-29T11:22:25.344 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.345+0000 7fc1a77fe640 1 -- 172.21.15.17:0/3814102075 >> v1:172.21.15.94:6789/0 conn(0x7fc1b01422b0 legacy=0x7fc1b01dec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:25.344 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.345+0000 7fc1a77fe640 1 -- 172.21.15.17:0/3814102075 >> v1:172.21.15.17:6789/0 conn(0x7fc1b00b17d0 legacy=0x7fc1b01502d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:25.344 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.345+0000 7fc1a77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:22:25.344 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.345+0000 7fc1a77fe640 1 monclient: found mon.c 2024-03-29T11:22:25.344 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.345+0000 7fc1a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:25.345 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.345+0000 7fc1a77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:22:25.345 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.345+0000 7fc1a77fe640 1 -- 172.21.15.17:0/3814102075 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b0131d20 con 0x7fc1b0142690 2024-03-29T11:22:25.345 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.345+0000 7fc1a77fe640 10 monclient: _finish_auth 0 2024-03-29T11:22:25.345 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.345+0000 7fc1a77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:22:25.345 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.345+0000 7fc1a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:25.345 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.345+0000 7fc1b53e9640 5 monclient: authenticate success, global_id 4379 2024-03-29T11:22:25.345 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.345+0000 7fc1b53e9640 10 monclient: _renew_subs 2024-03-29T11:22:25.345 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.345+0000 7fc1b53e9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:22:25.345 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.345+0000 7fc1b53e9640 1 -- 172.21.15.17:0/3814102075 --> v1:172.21.15.17:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1b0132240 con 0x7fc1b0142690 2024-03-29T11:22:25.345 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.345+0000 7fc1b53e9640 10 client.4379.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:25.345 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.345+0000 7fc1b53e9640 10 monclient: _renew_subs 2024-03-29T11:22:25.345 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.345+0000 7fc1b53e9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:22:25.345 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.345+0000 7fc1b53e9640 1 -- 172.21.15.17:0/3814102075 --> v1:172.21.15.17:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1b012ead0 con 0x7fc1b0142690 2024-03-29T11:22:25.345 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.345+0000 7fc1b53e9640 1 librados: init done 2024-03-29T11:22:25.345 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.346+0000 7fc1a77fe640 1 -- 172.21.15.17:0/3814102075 <== mon.2 v1:172.21.15.17:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc1a8003990 con 0x7fc1b0142690 2024-03-29T11:22:25.345 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.346+0000 7fc1a77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:25.346 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.346+0000 7fc1ad7fa640 4 set_mon_vals no callback set 2024-03-29T11:22:25.346 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.346+0000 7fc1a77fe640 1 -- 172.21.15.17:0/3814102075 <== mon.2 v1:172.21.15.17:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fc1a8003d60 con 0x7fc1b0142690 2024-03-29T11:22:25.346 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.346+0000 7fc1a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:25.346 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.346+0000 7fc1a77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:25.346 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.346+0000 7fc1a77fe640 10 monclient: dump: 2024-03-29T11:22:25.346 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:22:25.346 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:22:25.346 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:25.346 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:25.346 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:25.346 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:25.346 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:22:25.346 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:22:25.346 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:22:25.347 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:25.347 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.346+0000 7fc1b53e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:25.347 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.347+0000 7fc1b53e9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:25.347 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.347+0000 7fc1b53e9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:22:25.347 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.347+0000 7fc1b53e9640 1 -- 172.21.15.17:0/3814102075 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc194005080 con 0x7fc1b0142690 2024-03-29T11:22:25.347 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.348+0000 7fc1a77fe640 1 -- 172.21.15.17:0/3814102075 <== mon.2 v1:172.21.15.17:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7fc1a8003ec0 con 0x7fc1b0142690 2024-03-29T11:22:25.348 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.349+0000 7fc1a77fe640 1 -- 172.21.15.17:0/3814102075 <== mon.2 v1:172.21.15.17:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7fc1a8084e80 con 0x7fc1b0142690 2024-03-29T11:22:25.348 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.349+0000 7fc1aeffd640 10 monclient: get_auth_request con 0x7fc19006e110 auth_method 0 2024-03-29T11:22:25.348 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.349+0000 7fc1a77fe640 10 client.4379.objecter ms_dispatch 0x7fc1b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:25.348 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.349+0000 7fc1a77fe640 3 client.4379.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:25.348 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.349+0000 7fc1a77fe640 3 client.4379.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:25.348 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.349+0000 7fc1a77fe640 20 client.4379.objecter dump_active .. 0 homeless 2024-03-29T11:22:25.348 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.349+0000 7fc1a77fe640 10 client.4379.objecter ms_handle_connect 0x7fc19006e110 2024-03-29T11:22:25.349 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.357+0000 7fc1a77fe640 1 -- 172.21.15.17:0/3814102075 <== mon.2 v1:172.21.15.17:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc1a8050760 con 0x7fc1b0142690 2024-03-29T11:22:25.349 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.358+0000 7fc1a77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:25.349 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.358+0000 7fc1a77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:25.718 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.741+0000 7fc1b53e9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:25.718 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.741+0000 7fc1b53e9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:25.718 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.741+0000 7fc1b53e9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:22:25.718 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.741+0000 7fc1b53e9640 1 -- 172.21.15.17:0/3814102075 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc194005330 con 0x7fc1b0142690 2024-03-29T11:22:25.719 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.742+0000 7fc1a77fe640 1 -- 172.21.15.17:0/3814102075 <== mon.2 v1:172.21.15.17:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7fc1a8052690 con 0x7fc1b0142690 2024-03-29T11:22:25.719 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.742+0000 7fc1a77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:25.719 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.742+0000 7fc1a77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:25.719 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:22:25.719 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:25.723 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.745+0000 7fc1b53e9640 10 librados: watch_flush enter 2024-03-29T11:22:25.723 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.745+0000 7fc1b53e9640 10 librados: watch_flush exit 2024-03-29T11:22:25.723 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.746+0000 7fc1b53e9640 20 client.4379.objecter shutdown clearing up homeless session... 2024-03-29T11:22:25.723 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.746+0000 7fc1b53e9640 10 client.4379.objecter successfully canceled tick 2024-03-29T11:22:25.723 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.746+0000 7fc1b53e9640 1 -- 172.21.15.17:0/3814102075 >> v1:172.21.15.17:6816/3236185052 conn(0x7fc19006e110 legacy=0x7fc1900705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:25.723 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.746+0000 7fc1b53e9640 10 monclient: shutdown 2024-03-29T11:22:25.723 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.746+0000 7fc1b53e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:25.723 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.746+0000 7fc1b53e9640 1 -- 172.21.15.17:0/3814102075 >> v1:172.21.15.17:6790/0 conn(0x7fc1b0142690 legacy=0x7fc1b01e6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:25.723 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.746+0000 7fc1b53e9640 1 -- 172.21.15.17:0/3814102075 shutdown_connections 2024-03-29T11:22:25.723 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.746+0000 7fc1b53e9640 1 -- 172.21.15.17:0/3814102075 >> 172.21.15.17:0/3814102075 conn(0x7fc1b0066a00 msgr2=0x7fc1b013e370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:25.724 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.746+0000 7fc1b53e9640 1 -- 172.21.15.17:0/3814102075 shutdown_connections 2024-03-29T11:22:25.724 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.746+0000 7fc1b53e9640 1 -- 172.21.15.17:0/3814102075 wait complete. 2024-03-29T11:22:25.724 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:25.746+0000 7fc1b53e9640 1 librados: shutdown 2024-03-29T11:22:25.738 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:32.544 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:32.684 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.707+0000 7f53ca086640 10 monclient: get_monmap_and_config 2024-03-29T11:22:32.684 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.707+0000 7f53ca086640 10 monclient: build_initial_monmap 2024-03-29T11:22:32.684 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.707+0000 7f53ca086640 10 monclient: monmap: 2024-03-29T11:22:32.684 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:22:32.684 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:22:32.684 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:22:32.708645+0000 2024-03-29T11:22:32.684 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:22:32.708645+0000 2024-03-29T11:22:32.684 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:32.685 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:32.685 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:22:32.685 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:22:32.685 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:22:32.685 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:32.685 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.708+0000 7f53ca086640 1 Processor -- start 2024-03-29T11:22:32.685 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.708+0000 7f53ca086640 1 -- start start 2024-03-29T11:22:32.685 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.709+0000 7f53ca086640 10 monclient: init 2024-03-29T11:22:32.686 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.709+0000 7f53ca086640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:32.686 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.709+0000 7f53ca086640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:22:32.686 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.709+0000 7f53ca086640 10 monclient(hunting): picked mon.noname-b con 0x7f53c41457f0 addr v1:172.21.15.17:6790/0 2024-03-29T11:22:32.686 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.710+0000 7f53c37fe640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f53c41457f0 0x7f53c413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:34456/0 (socket says 172.21.15.17:34456) 2024-03-29T11:22:32.687 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.710+0000 7f53c37fe640 1 -- 172.21.15.17:0/3059810175 learned_addr learned my addr 172.21.15.17:0/3059810175 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:22:32.687 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.710+0000 7f53ca086640 10 monclient(hunting): picked mon.noname-a con 0x7f53c413f650 addr v1:172.21.15.17:6789/0 2024-03-29T11:22:32.687 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.711+0000 7f53ca086640 10 monclient(hunting): picked mon.noname-c con 0x7f53c40a9d70 addr v1:172.21.15.94:6789/0 2024-03-29T11:22:32.688 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.711+0000 7f53ca086640 1 -- 172.21.15.17:0/3059810175 --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53c40aaef0 con 0x7f53c413f650 2024-03-29T11:22:32.688 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.711+0000 7f53ca086640 1 -- 172.21.15.17:0/3059810175 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53c4145bd0 con 0x7f53c40a9d70 2024-03-29T11:22:32.688 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.711+0000 7f53ca086640 1 -- 172.21.15.17:0/3059810175 --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53c4145eb0 con 0x7f53c41457f0 2024-03-29T11:22:32.688 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.711+0000 7f53ca086640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:32.688 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.711+0000 7f53ca086640 10 monclient(hunting): authenticate will time out at 1058.916023s 2024-03-29T11:22:32.691 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.713+0000 7f53c27fc640 1 -- 172.21.15.17:0/3059810175 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3210486431 0 0) 0x7f53c4145bd0 con 0x7f53c40a9d70 2024-03-29T11:22:32.691 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.713+0000 7f53c27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:32.691 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.713+0000 7f53c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:32.691 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.713+0000 7f53c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:32.691 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.713+0000 7f53c27fc640 10 monclient(hunting): my global_id is 4360 2024-03-29T11:22:32.691 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.713+0000 7f53c27fc640 1 -- 172.21.15.17:0/3059810175 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53ac003520 con 0x7f53c40a9d70 2024-03-29T11:22:32.691 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.713+0000 7f53c27fc640 1 -- 172.21.15.17:0/3059810175 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 66967021 0 0) 0x7f53c40aaef0 con 0x7f53c413f650 2024-03-29T11:22:32.691 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.713+0000 7f53c27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:32.691 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.713+0000 7f53c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:32.692 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.713+0000 7f53c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:32.692 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.713+0000 7f53c27fc640 10 monclient(hunting): my global_id is 4359 2024-03-29T11:22:32.692 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.713+0000 7f53c27fc640 1 -- 172.21.15.17:0/3059810175 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53c4145bd0 con 0x7f53c413f650 2024-03-29T11:22:32.692 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.713+0000 7f53c27fc640 1 -- 172.21.15.17:0/3059810175 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2598713008 0 0) 0x7f53c4145eb0 con 0x7f53c41457f0 2024-03-29T11:22:32.692 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.713+0000 7f53c27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:32.692 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.713+0000 7f53c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:32.692 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.713+0000 7f53c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:32.692 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.713+0000 7f53c27fc640 10 monclient(hunting): my global_id is 4382 2024-03-29T11:22:32.692 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.713+0000 7f53c27fc640 1 -- 172.21.15.17:0/3059810175 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53c40aaef0 con 0x7f53c41457f0 2024-03-29T11:22:32.692 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.713+0000 7f53c27fc640 1 -- 172.21.15.17:0/3059810175 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4183532142 0 0) 0x7f53c4145bd0 con 0x7f53c413f650 2024-03-29T11:22:32.692 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.713+0000 7f53c27fc640 1 -- 172.21.15.17:0/3059810175 >> v1:172.21.15.17:6790/0 conn(0x7f53c41457f0 legacy=0x7f53c413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:32.692 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.713+0000 7f53c27fc640 1 -- 172.21.15.17:0/3059810175 >> v1:172.21.15.94:6789/0 conn(0x7f53c40a9d70 legacy=0x7f53c414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:32.692 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.713+0000 7f53c27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:22:32.692 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.713+0000 7f53c27fc640 1 monclient: found mon.noname-a 2024-03-29T11:22:32.692 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.713+0000 7f53c27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:32.692 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.713+0000 7f53c27fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.17:6789/0 2024-03-29T11:22:32.693 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.713+0000 7f53c27fc640 1 -- 172.21.15.17:0/3059810175 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53c4146340 con 0x7f53c413f650 2024-03-29T11:22:32.693 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.713+0000 7f53c27fc640 10 monclient: _finish_auth 0 2024-03-29T11:22:32.693 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.713+0000 7f53c27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:22:32.693 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.713+0000 7f53c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:32.693 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.713+0000 7f53c27fc640 1 -- 172.21.15.17:0/3059810175 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f53b4001670 con 0x7f53c413f650 2024-03-29T11:22:32.693 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.713+0000 7f53c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:32.693 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.713+0000 7f53c27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:32.693 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.713+0000 7f53c27fc640 10 monclient: dump: 2024-03-29T11:22:32.693 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:22:32.693 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:22:32.693 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:32.693 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:32.693 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:32.693 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:32.693 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:22:32.693 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:22:32.693 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:22:32.694 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:32.694 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.713+0000 7f53ca086640 5 monclient: authenticate success, global_id 4359 2024-03-29T11:22:32.694 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.713+0000 7f53ca086640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:32.694 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.713+0000 7f53c27fc640 1 -- 172.21.15.17:0/3059810175 <== mon.0 v1:172.21.15.17:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53b40017d0 con 0x7f53c413f650 2024-03-29T11:22:32.694 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.713+0000 7f53c27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:32.694 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.713+0000 7f53c27fc640 1 -- 172.21.15.17:0/3059810175 <== mon.0 v1:172.21.15.17:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f53b4001be0 con 0x7f53c413f650 2024-03-29T11:22:32.694 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.713+0000 7f53c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:32.694 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.713+0000 7f53c27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:32.694 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.713+0000 7f53c27fc640 10 monclient: dump: 2024-03-29T11:22:32.694 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:22:32.694 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:22:32.694 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:32.694 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:32.694 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:32.695 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:32.695 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:22:32.695 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:22:32.695 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:22:32.695 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:32.695 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.713+0000 7f53ca086640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:32.695 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.713+0000 7f53ca086640 4 set_mon_vals no callback set 2024-03-29T11:22:32.695 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.714+0000 7f53ca086640 10 monclient: shutdown 2024-03-29T11:22:32.695 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.714+0000 7f53ca086640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:32.695 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.714+0000 7f53ca086640 1 -- 172.21.15.17:0/3059810175 >> v1:172.21.15.17:6789/0 conn(0x7f53c413f650 legacy=0x7f53c40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:32.695 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.714+0000 7f53ca086640 1 -- 172.21.15.17:0/3059810175 shutdown_connections 2024-03-29T11:22:32.695 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.714+0000 7f53ca086640 1 -- 172.21.15.17:0/3059810175 >> 172.21.15.17:0/3059810175 conn(0x7f53c4066a00 msgr2=0x7f53c4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:32.695 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.714+0000 7f53ca086640 1 -- 172.21.15.17:0/3059810175 shutdown_connections 2024-03-29T11:22:32.695 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.714+0000 7f53ca086640 1 -- 172.21.15.17:0/3059810175 wait complete. 2024-03-29T11:22:32.703 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.715+0000 7f53ca086640 10 monclient: build_initial_monmap 2024-03-29T11:22:32.703 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.715+0000 7f53ca086640 10 monclient: monmap: 2024-03-29T11:22:32.703 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:22:32.703 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:32.703 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:22:32.703 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:22:32.703 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:32.703 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:32.703 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:22:32.703 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:22:32.704 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:22:32.704 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:32.704 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.716+0000 7f53ca086640 1 librados: starting msgr at 2024-03-29T11:22:32.704 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.716+0000 7f53ca086640 1 librados: starting objecter 2024-03-29T11:22:32.704 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.716+0000 7f53ca086640 1 Processor -- start 2024-03-29T11:22:32.704 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.716+0000 7f53ca086640 1 -- start start 2024-03-29T11:22:32.704 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.716+0000 7f53ca086640 1 librados: setting wanted keys 2024-03-29T11:22:32.704 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.716+0000 7f53ca086640 1 librados: calling monclient init 2024-03-29T11:22:32.704 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.716+0000 7f53ca086640 10 monclient: init 2024-03-29T11:22:32.704 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.717+0000 7f53ca086640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:32.704 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.717+0000 7f53ca086640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:32.704 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.717+0000 7f53ca086640 10 monclient(hunting): picked mon.noname-c con 0x7f53c40a9d70 addr v1:172.21.15.17:6790/0 2024-03-29T11:22:32.704 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.717+0000 7f53c37fe640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f53c40a9d70 0x7f53c4137710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:35424/0 (socket says 172.21.15.17:35424) 2024-03-29T11:22:32.705 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.717+0000 7f53c37fe640 1 -- 172.21.15.17:0/1867666621 learned_addr learned my addr 172.21.15.17:0/1867666621 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:22:32.705 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.717+0000 7f53abfff640 10 client.?.objecter ms_handle_connect 0x7f53c40a9d70 2024-03-29T11:22:32.705 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.717+0000 7f53abfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:32.705 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.717+0000 7f53ca086640 10 monclient(hunting): picked mon.noname-b con 0x7f53c41457f0 addr v1:172.21.15.94:6789/0 2024-03-29T11:22:32.705 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.717+0000 7f53ca086640 10 monclient(hunting): picked mon.noname-a con 0x7f53c413b060 addr v1:172.21.15.17:6789/0 2024-03-29T11:22:32.705 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.718+0000 7f53abfff640 10 client.?.objecter ms_handle_connect 0x7f53c41457f0 2024-03-29T11:22:32.705 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.718+0000 7f53abfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:32.705 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.718+0000 7f53ca086640 1 -- 172.21.15.17:0/1867666621 --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53c406f480 con 0x7f53c413b060 2024-03-29T11:22:32.705 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.718+0000 7f53ca086640 1 -- 172.21.15.17:0/1867666621 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53c406de90 con 0x7f53c41457f0 2024-03-29T11:22:32.705 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.718+0000 7f53ca086640 1 -- 172.21.15.17:0/1867666621 --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53c406f6b0 con 0x7f53c40a9d70 2024-03-29T11:22:32.705 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.718+0000 7f53ca086640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:32.705 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.718+0000 7f53ca086640 10 monclient(hunting): authenticate will time out at 1058.923548s 2024-03-29T11:22:32.705 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.718+0000 7f53abfff640 1 -- 172.21.15.17:0/1867666621 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4015734714 0 0) 0x7f53c406de90 con 0x7f53c41457f0 2024-03-29T11:22:32.705 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.719+0000 7f53abfff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:32.705 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.719+0000 7f53abfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:32.706 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.719+0000 7f53abfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:32.706 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.719+0000 7f53abfff640 10 monclient(hunting): my global_id is 4363 2024-03-29T11:22:32.706 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.719+0000 7f53abfff640 1 -- 172.21.15.17:0/1867666621 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53b0004eb0 con 0x7f53c41457f0 2024-03-29T11:22:32.706 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.719+0000 7f53abfff640 1 -- 172.21.15.17:0/1867666621 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3685374928 0 0) 0x7f53c406f6b0 con 0x7f53c40a9d70 2024-03-29T11:22:32.706 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.719+0000 7f53abfff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:32.706 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.719+0000 7f53abfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:32.706 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.719+0000 7f53abfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:32.706 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.719+0000 7f53abfff640 10 monclient(hunting): my global_id is 4385 2024-03-29T11:22:32.706 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.719+0000 7f53abfff640 1 -- 172.21.15.17:0/1867666621 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53c406de90 con 0x7f53c40a9d70 2024-03-29T11:22:32.706 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.719+0000 7f53abfff640 1 -- 172.21.15.17:0/1867666621 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1352150884 0 0) 0x7f53c406de90 con 0x7f53c40a9d70 2024-03-29T11:22:32.706 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.719+0000 7f53abfff640 1 -- 172.21.15.17:0/1867666621 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53c406f6b0 con 0x7f53c40a9d70 2024-03-29T11:22:32.706 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.719+0000 7f53abfff640 1 -- 172.21.15.17:0/1867666621 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f53b8001850 con 0x7f53c40a9d70 2024-03-29T11:22:32.706 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.719+0000 7f53abfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:32.707 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.719+0000 7f53abfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:32.707 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.719+0000 7f53abfff640 10 monclient(hunting): dump: 2024-03-29T11:22:32.707 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:22:32.707 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:22:32.707 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:32.707 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:32.707 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:32.707 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:32.707 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:22:32.707 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:22:32.707 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:22:32.707 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:32.707 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.719+0000 7f53abfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:32.707 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.720+0000 7f53abfff640 1 -- 172.21.15.17:0/1867666621 <== mon.2 v1:172.21.15.17:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1007571599 0 0) 0x7f53c406f6b0 con 0x7f53c40a9d70 2024-03-29T11:22:32.707 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.720+0000 7f53abfff640 1 -- 172.21.15.17:0/1867666621 >> v1:172.21.15.94:6789/0 conn(0x7f53c41457f0 legacy=0x7f53c41ded90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:32.708 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.720+0000 7f53abfff640 1 -- 172.21.15.17:0/1867666621 >> v1:172.21.15.17:6789/0 conn(0x7f53c413b060 legacy=0x7f53c41df4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:32.708 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.721+0000 7f53abfff640 10 monclient: _finish_hunting 0 2024-03-29T11:22:32.708 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.721+0000 7f53abfff640 1 monclient: found mon.c 2024-03-29T11:22:32.708 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.721+0000 7f53abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:32.708 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.721+0000 7f53abfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:22:32.708 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.721+0000 7f53abfff640 1 -- 172.21.15.17:0/1867666621 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53c4138cc0 con 0x7f53c40a9d70 2024-03-29T11:22:32.708 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.721+0000 7f53abfff640 10 monclient: _finish_auth 0 2024-03-29T11:22:32.708 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.721+0000 7f53abfff640 10 monclient: _check_auth_tickets 2024-03-29T11:22:32.708 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.721+0000 7f53abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:32.708 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.721+0000 7f53ca086640 5 monclient: authenticate success, global_id 4385 2024-03-29T11:22:32.708 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.721+0000 7f53ca086640 10 monclient: _renew_subs 2024-03-29T11:22:32.708 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.721+0000 7f53ca086640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:22:32.708 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.721+0000 7f53ca086640 1 -- 172.21.15.17:0/1867666621 --> v1:172.21.15.17:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53c4138f10 con 0x7f53c40a9d70 2024-03-29T11:22:32.708 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.721+0000 7f53ca086640 10 client.4385.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:32.708 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.721+0000 7f53ca086640 10 monclient: _renew_subs 2024-03-29T11:22:32.709 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.721+0000 7f53ca086640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:22:32.709 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.721+0000 7f53ca086640 1 -- 172.21.15.17:0/1867666621 --> v1:172.21.15.17:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f53c413bb70 con 0x7f53c40a9d70 2024-03-29T11:22:32.709 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.721+0000 7f53ca086640 1 librados: init done 2024-03-29T11:22:32.709 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.721+0000 7f53abfff640 1 -- 172.21.15.17:0/1867666621 <== mon.2 v1:172.21.15.17:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53b8001d60 con 0x7f53c40a9d70 2024-03-29T11:22:32.709 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.721+0000 7f53abfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:32.709 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.721+0000 7f53abfff640 1 -- 172.21.15.17:0/1867666621 <== mon.2 v1:172.21.15.17:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f53b8004980 con 0x7f53c40a9d70 2024-03-29T11:22:32.709 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.721+0000 7f53abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:32.709 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.721+0000 7f53abfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:32.709 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.721+0000 7f53abfff640 10 monclient: dump: 2024-03-29T11:22:32.709 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:22:32.709 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:22:32.709 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:32.709 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:32.709 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:32.709 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:32.710 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:22:32.710 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:22:32.710 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:22:32.710 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:32.710 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.721+0000 7f53c1ffb640 4 set_mon_vals no callback set 2024-03-29T11:22:32.710 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.723+0000 7f53abfff640 1 -- 172.21.15.17:0/1867666621 <== mon.2 v1:172.21.15.17:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f53b8004ae0 con 0x7f53c40a9d70 2024-03-29T11:22:32.710 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.723+0000 7f53ca086640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:32.710 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.723+0000 7f53ca086640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:32.710 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.723+0000 7f53ca086640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:22:32.710 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.723+0000 7f53ca086640 1 -- 172.21.15.17:0/1867666621 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53a4005080 con 0x7f53c40a9d70 2024-03-29T11:22:32.710 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.724+0000 7f53abfff640 1 -- 172.21.15.17:0/1867666621 <== mon.2 v1:172.21.15.17:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f53b8084a70 con 0x7f53c40a9d70 2024-03-29T11:22:32.710 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.724+0000 7f53abfff640 10 client.4385.objecter ms_dispatch 0x7f53c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:32.710 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.724+0000 7f53abfff640 3 client.4385.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:32.710 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.724+0000 7f53abfff640 3 client.4385.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:32.711 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.724+0000 7f53c2ffd640 10 monclient: get_auth_request con 0x7f53b006dd60 auth_method 0 2024-03-29T11:22:32.711 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.724+0000 7f53abfff640 20 client.4385.objecter dump_active .. 0 homeless 2024-03-29T11:22:32.711 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.724+0000 7f53abfff640 10 client.4385.objecter ms_handle_connect 0x7f53b006dd60 2024-03-29T11:22:32.711 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.730+0000 7f53abfff640 1 -- 172.21.15.17:0/1867666621 <== mon.2 v1:172.21.15.17:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f53b8050350 con 0x7f53c40a9d70 2024-03-29T11:22:32.711 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.730+0000 7f53abfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:32.712 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:32.730+0000 7f53abfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:33.021 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:33.044+0000 7f53ca086640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:33.021 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:33.044+0000 7f53ca086640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:33.021 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:33.044+0000 7f53ca086640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:22:33.021 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:33.044+0000 7f53ca086640 1 -- 172.21.15.17:0/1867666621 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53a4005330 con 0x7f53c40a9d70 2024-03-29T11:22:33.022 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:33.045+0000 7f53abfff640 1 -- 172.21.15.17:0/1867666621 <== mon.2 v1:172.21.15.17:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f53b8052280 con 0x7f53c40a9d70 2024-03-29T11:22:33.022 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:33.045+0000 7f53abfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:33.022 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:33.045+0000 7f53abfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:33.022 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:22:33.022 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:33.025 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:33.048+0000 7f53ca086640 10 librados: watch_flush enter 2024-03-29T11:22:33.025 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:33.048+0000 7f53ca086640 10 librados: watch_flush exit 2024-03-29T11:22:33.025 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:33.048+0000 7f53ca086640 20 client.4385.objecter shutdown clearing up homeless session... 2024-03-29T11:22:33.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:33.048+0000 7f53ca086640 10 client.4385.objecter successfully canceled tick 2024-03-29T11:22:33.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:33.049+0000 7f53ca086640 1 -- 172.21.15.17:0/1867666621 >> v1:172.21.15.17:6816/3236185052 conn(0x7f53b006dd60 legacy=0x7f53b0070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:33.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:33.049+0000 7f53ca086640 10 monclient: shutdown 2024-03-29T11:22:33.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:33.049+0000 7f53ca086640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:33.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:33.049+0000 7f53ca086640 1 -- 172.21.15.17:0/1867666621 >> v1:172.21.15.17:6790/0 conn(0x7f53c40a9d70 legacy=0x7f53c4137710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:33.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:33.049+0000 7f53ca086640 1 -- 172.21.15.17:0/1867666621 shutdown_connections 2024-03-29T11:22:33.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:33.049+0000 7f53ca086640 1 -- 172.21.15.17:0/1867666621 >> 172.21.15.17:0/1867666621 conn(0x7f53c4066a00 msgr2=0x7f53c40b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:33.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:33.049+0000 7f53ca086640 1 -- 172.21.15.17:0/1867666621 shutdown_connections 2024-03-29T11:22:33.027 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:33.049+0000 7f53ca086640 1 -- 172.21.15.17:0/1867666621 wait complete. 2024-03-29T11:22:33.027 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:33.050+0000 7f53ca086640 1 librados: shutdown 2024-03-29T11:22:33.040 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:39.846 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:39.997 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.020+0000 7feba099c640 10 monclient: get_monmap_and_config 2024-03-29T11:22:39.998 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.020+0000 7feba099c640 10 monclient: build_initial_monmap 2024-03-29T11:22:39.998 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.020+0000 7feba099c640 10 monclient: monmap: 2024-03-29T11:22:39.998 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:22:39.998 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:22:39.998 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:22:40.021729+0000 2024-03-29T11:22:39.998 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:22:40.021729+0000 2024-03-29T11:22:39.998 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:39.998 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:39.998 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:22:39.998 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:22:39.998 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:22:39.998 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:40.001 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.024+0000 7feba099c640 1 Processor -- start 2024-03-29T11:22:40.004 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.024+0000 7feba099c640 1 -- start start 2024-03-29T11:22:40.007 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.024+0000 7feba099c640 10 monclient: init 2024-03-29T11:22:40.007 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.025+0000 7feba099c640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:40.007 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.025+0000 7feba099c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:22:40.007 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.025+0000 7feba099c640 10 monclient(hunting): picked mon.noname-a con 0x7feb940389e0 addr v1:172.21.15.17:6789/0 2024-03-29T11:22:40.007 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.025+0000 7feba099c640 10 monclient(hunting): picked mon.noname-c con 0x7feb94038dc0 addr v1:172.21.15.94:6789/0 2024-03-29T11:22:40.007 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.025+0000 7feba099c640 10 monclient(hunting): picked mon.noname-b con 0x7feb94030660 addr v1:172.21.15.17:6790/0 2024-03-29T11:22:40.007 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.025+0000 7feba099c640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb940c54c0 con 0x7feb940389e0 2024-03-29T11:22:40.007 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.025+0000 7feba099c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb94036430 con 0x7feb94038dc0 2024-03-29T11:22:40.007 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.025+0000 7feba099c640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb94039840 con 0x7feb94030660 2024-03-29T11:22:40.007 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.025+0000 7feba099c640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:40.007 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.025+0000 7feba099c640 10 monclient(hunting): authenticate will time out at 1066.230837s 2024-03-29T11:22:40.008 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.028+0000 7feb9a59c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7feb94038dc0 0x7feb940cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:37935/0 (socket says 172.21.15.17:37935) 2024-03-29T11:22:40.008 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.028+0000 7feb9a59c640 1 -- 172.21.15.17:0/4011209853 learned_addr learned my addr 172.21.15.17:0/4011209853 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:22:40.008 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.028+0000 7feb99d9b640 1 -- 172.21.15.17:0/4011209853 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2447120324 0 0) 0x7feb940c54c0 con 0x7feb940389e0 2024-03-29T11:22:40.008 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.028+0000 7feb99d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:22:40.008 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.028+0000 7feb99d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:40.008 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.028+0000 7feb99d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:40.008 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.028+0000 7feb99d9b640 10 monclient(hunting): my global_id is 4362 2024-03-29T11:22:40.008 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.028+0000 7feb99d9b640 1 -- 172.21.15.17:0/4011209853 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb84003520 con 0x7feb940389e0 2024-03-29T11:22:40.010 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.031+0000 7feb99d9b640 1 -- 172.21.15.17:0/4011209853 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1182498517 0 0) 0x7feb84003520 con 0x7feb940389e0 2024-03-29T11:22:40.011 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.031+0000 7feb99d9b640 1 -- 172.21.15.17:0/4011209853 >> v1:172.21.15.17:6790/0 conn(0x7feb94030660 legacy=0x7feb940d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:40.011 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.031+0000 7feb99d9b640 1 -- 172.21.15.17:0/4011209853 >> v1:172.21.15.94:6789/0 conn(0x7feb94038dc0 legacy=0x7feb940cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:40.011 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.031+0000 7feb99d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:22:40.011 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.031+0000 7feb99d9b640 1 monclient: found mon.noname-a 2024-03-29T11:22:40.011 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.031+0000 7feb99d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:40.011 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.031+0000 7feb99d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.17:6789/0 2024-03-29T11:22:40.011 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.031+0000 7feb99d9b640 1 -- 172.21.15.17:0/4011209853 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb94035690 con 0x7feb940389e0 2024-03-29T11:22:40.011 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.031+0000 7feb99d9b640 10 monclient: _finish_auth 0 2024-03-29T11:22:40.011 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.031+0000 7feb99d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:22:40.011 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.031+0000 7feb99d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:40.011 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.031+0000 7feba099c640 5 monclient: authenticate success, global_id 4362 2024-03-29T11:22:40.011 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.031+0000 7feba099c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:40.011 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.032+0000 7feb99d9b640 1 -- 172.21.15.17:0/4011209853 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7feb90001690 con 0x7feb940389e0 2024-03-29T11:22:40.012 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.032+0000 7feb99d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:40.012 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.032+0000 7feb99d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:40.012 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.032+0000 7feb99d9b640 10 monclient: dump: 2024-03-29T11:22:40.012 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:22:40.012 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:22:40.012 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:40.012 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:40.012 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:40.012 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:40.012 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:22:40.012 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:22:40.012 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:22:40.012 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:40.012 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.032+0000 7feba099c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:40.013 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.032+0000 7feb99d9b640 1 -- 172.21.15.17:0/4011209853 <== mon.0 v1:172.21.15.17:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb900018f0 con 0x7feb940389e0 2024-03-29T11:22:40.013 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.032+0000 7feb99d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:40.013 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.032+0000 7feba099c640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:40.013 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.032+0000 7feb99d9b640 1 -- 172.21.15.17:0/4011209853 <== mon.0 v1:172.21.15.17:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7feb90001690 con 0x7feb940389e0 2024-03-29T11:22:40.013 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.032+0000 7feb99d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:40.013 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.032+0000 7feb99d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:40.013 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.032+0000 7feb99d9b640 10 monclient: dump: 2024-03-29T11:22:40.013 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:22:40.013 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:22:40.013 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:40.013 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:40.013 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:40.013 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:40.013 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:22:40.013 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:22:40.014 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:22:40.014 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:40.014 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.032+0000 7feba099c640 4 set_mon_vals no callback set 2024-03-29T11:22:40.014 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.033+0000 7feba099c640 10 monclient: shutdown 2024-03-29T11:22:40.014 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.033+0000 7feba099c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:40.014 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.033+0000 7feba099c640 1 -- 172.21.15.17:0/4011209853 >> v1:172.21.15.17:6789/0 conn(0x7feb940389e0 legacy=0x7feb94033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:40.014 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.033+0000 7feba099c640 1 -- 172.21.15.17:0/4011209853 shutdown_connections 2024-03-29T11:22:40.014 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.033+0000 7feba099c640 1 -- 172.21.15.17:0/4011209853 >> 172.21.15.17:0/4011209853 conn(0x7feb94005c00 msgr2=0x7feb94006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:40.014 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.033+0000 7feba099c640 1 -- 172.21.15.17:0/4011209853 shutdown_connections 2024-03-29T11:22:40.014 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.033+0000 7feba099c640 1 -- 172.21.15.17:0/4011209853 wait complete. 2024-03-29T11:22:40.016 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.034+0000 7feba099c640 10 monclient: build_initial_monmap 2024-03-29T11:22:40.016 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.034+0000 7feba099c640 10 monclient: monmap: 2024-03-29T11:22:40.016 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:22:40.016 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:40.016 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:22:40.016 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:22:40.016 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:40.016 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:40.016 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:22:40.016 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:22:40.016 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:22:40.016 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:40.017 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.035+0000 7feba099c640 1 librados: starting msgr at 2024-03-29T11:22:40.017 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.035+0000 7feba099c640 1 librados: starting objecter 2024-03-29T11:22:40.017 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.035+0000 7feba099c640 1 Processor -- start 2024-03-29T11:22:40.017 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.035+0000 7feba099c640 1 -- start start 2024-03-29T11:22:40.017 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.035+0000 7feba099c640 1 librados: setting wanted keys 2024-03-29T11:22:40.017 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.035+0000 7feba099c640 1 librados: calling monclient init 2024-03-29T11:22:40.017 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.035+0000 7feba099c640 10 monclient: init 2024-03-29T11:22:40.017 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.036+0000 7feba099c640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:40.017 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.036+0000 7feba099c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:22:40.017 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.037+0000 7feba099c640 10 monclient(hunting): picked mon.noname-b con 0x7feb94030660 addr v1:172.21.15.94:6789/0 2024-03-29T11:22:40.017 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.037+0000 7feba099c640 10 monclient(hunting): picked mon.noname-c con 0x7feb940389e0 addr v1:172.21.15.17:6790/0 2024-03-29T11:22:40.017 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.037+0000 7feba099c640 10 monclient(hunting): picked mon.noname-a con 0x7feb940c2c50 addr v1:172.21.15.17:6789/0 2024-03-29T11:22:40.017 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.038+0000 7feba099c640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb940a51e0 con 0x7feb940c2c50 2024-03-29T11:22:40.017 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.038+0000 7feba099c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb94160260 con 0x7feb94030660 2024-03-29T11:22:40.018 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.038+0000 7feba099c640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb940c29c0 con 0x7feb940389e0 2024-03-29T11:22:40.018 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.038+0000 7feba099c640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:40.018 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.038+0000 7feba099c640 10 monclient(hunting): authenticate will time out at 1066.242921s 2024-03-29T11:22:40.020 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.040+0000 7feb9ad9d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7feb94030660 0x7feb9415fb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:38015/0 (socket says 172.21.15.17:38015) 2024-03-29T11:22:40.020 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.040+0000 7feb9ad9d640 1 -- 172.21.15.17:0/3867930512 learned_addr learned my addr 172.21.15.17:0/3867930512 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:22:40.020 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.040+0000 7feb837fe640 10 client.?.objecter ms_handle_connect 0x7feb94030660 2024-03-29T11:22:40.020 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.040+0000 7feb837fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:40.020 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.040+0000 7feb837fe640 1 -- 172.21.15.17:0/3867930512 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3984973082 0 0) 0x7feb94160260 con 0x7feb94030660 2024-03-29T11:22:40.020 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.040+0000 7feb837fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:40.020 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.040+0000 7feb837fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:40.020 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.041+0000 7feb837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:40.020 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.041+0000 7feb837fe640 10 monclient(hunting): my global_id is 4369 2024-03-29T11:22:40.021 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.041+0000 7feb837fe640 1 -- 172.21.15.17:0/3867930512 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb88004eb0 con 0x7feb94030660 2024-03-29T11:22:40.021 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.041+0000 7feb837fe640 1 -- 172.21.15.17:0/3867930512 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 771878839 0 0) 0x7feb88004eb0 con 0x7feb94030660 2024-03-29T11:22:40.021 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.041+0000 7feb837fe640 1 -- 172.21.15.17:0/3867930512 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb94160260 con 0x7feb94030660 2024-03-29T11:22:40.021 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.041+0000 7feb837fe640 1 -- 172.21.15.17:0/3867930512 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7feb90001690 con 0x7feb94030660 2024-03-29T11:22:40.021 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.041+0000 7feb837fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:40.021 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.042+0000 7feb837fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:40.021 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.042+0000 7feb837fe640 10 monclient(hunting): dump: 2024-03-29T11:22:40.021 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:22:40.021 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:22:40.021 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:40.021 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:40.021 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:40.021 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:40.021 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:22:40.021 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:22:40.022 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:22:40.022 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:40.022 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.042+0000 7feb837fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:40.022 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.042+0000 7feb837fe640 1 -- 172.21.15.17:0/3867930512 <== mon.1 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3836613044 0 0) 0x7feb94160260 con 0x7feb94030660 2024-03-29T11:22:40.022 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.042+0000 7feb837fe640 1 -- 172.21.15.17:0/3867930512 >> v1:172.21.15.17:6790/0 conn(0x7feb940389e0 legacy=0x7feb940c1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:40.022 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.042+0000 7feb837fe640 1 -- 172.21.15.17:0/3867930512 >> v1:172.21.15.17:6789/0 conn(0x7feb940c2c50 legacy=0x7feb940c22b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:40.024 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.044+0000 7feb837fe640 10 monclient: _finish_hunting 0 2024-03-29T11:22:40.025 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.044+0000 7feb837fe640 1 monclient: found mon.b 2024-03-29T11:22:40.025 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.044+0000 7feb837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:40.025 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.044+0000 7feb837fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:22:40.025 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.044+0000 7feb837fe640 1 -- 172.21.15.17:0/3867930512 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb9402f450 con 0x7feb94030660 2024-03-29T11:22:40.025 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.044+0000 7feb837fe640 10 monclient: _finish_auth 0 2024-03-29T11:22:40.025 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.045+0000 7feb837fe640 10 monclient: _check_auth_tickets 2024-03-29T11:22:40.025 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.045+0000 7feb837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:40.025 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.045+0000 7feb837fe640 10 client.?.objecter ms_handle_connect 0x7feb940389e0 2024-03-29T11:22:40.025 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.045+0000 7feb837fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:40.025 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.045+0000 7feb837fe640 1 -- 172.21.15.17:0/3867930512 <== mon.1 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb90001cc0 con 0x7feb94030660 2024-03-29T11:22:40.025 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.045+0000 7feb837fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:40.025 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.045+0000 7feb9959a640 4 set_mon_vals no callback set 2024-03-29T11:22:40.025 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.046+0000 7feb837fe640 1 -- 172.21.15.17:0/3867930512 <== mon.1 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7feb90004040 con 0x7feb94030660 2024-03-29T11:22:40.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.046+0000 7feb837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:40.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.046+0000 7feb837fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:40.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.046+0000 7feb837fe640 10 monclient: dump: 2024-03-29T11:22:40.026 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:22:40.026 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:22:40.026 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:40.026 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:40.026 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:40.026 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:40.026 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:22:40.026 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:22:40.026 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:22:40.026 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:40.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.046+0000 7feba099c640 5 monclient: authenticate success, global_id 4369 2024-03-29T11:22:40.027 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.046+0000 7feba099c640 10 monclient: _renew_subs 2024-03-29T11:22:40.027 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.046+0000 7feba099c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:22:40.027 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.046+0000 7feba099c640 1 -- 172.21.15.17:0/3867930512 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb9402e770 con 0x7feb94030660 2024-03-29T11:22:40.027 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.046+0000 7feba099c640 10 client.4369.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:40.027 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.046+0000 7feba099c640 10 monclient: _renew_subs 2024-03-29T11:22:40.027 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.046+0000 7feba099c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:22:40.027 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.046+0000 7feba099c640 1 -- 172.21.15.17:0/3867930512 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feb9402f1c0 con 0x7feb94030660 2024-03-29T11:22:40.027 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.046+0000 7feba099c640 1 librados: init done 2024-03-29T11:22:40.027 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.050+0000 7feba099c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:40.028 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.050+0000 7feba099c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:40.028 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.050+0000 7feba099c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:22:40.028 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.050+0000 7feba099c640 1 -- 172.21.15.17:0/3867930512 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb94007b90 con 0x7feb94030660 2024-03-29T11:22:40.028 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.050+0000 7feb837fe640 1 -- 172.21.15.17:0/3867930512 <== mon.1 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7feb900041a0 con 0x7feb94030660 2024-03-29T11:22:40.028 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.051+0000 7feb9a59c640 10 monclient: get_auth_request con 0x7feb8806dca0 auth_method 0 2024-03-29T11:22:40.028 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.051+0000 7feb837fe640 1 -- 172.21.15.17:0/3867930512 <== mon.1 v1:172.21.15.94:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7feb90084dd0 con 0x7feb94030660 2024-03-29T11:22:40.028 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.051+0000 7feb837fe640 10 client.4369.objecter ms_dispatch 0x7feb9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:40.029 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.051+0000 7feb837fe640 3 client.4369.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:40.029 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.051+0000 7feb837fe640 3 client.4369.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:40.029 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.051+0000 7feb837fe640 20 client.4369.objecter dump_active .. 0 homeless 2024-03-29T11:22:40.029 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.052+0000 7feb837fe640 10 client.4369.objecter ms_handle_connect 0x7feb8806dca0 2024-03-29T11:22:40.036 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.059+0000 7feb837fe640 1 -- 172.21.15.17:0/3867930512 <== mon.1 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feb900506b0 con 0x7feb94030660 2024-03-29T11:22:40.036 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.059+0000 7feb837fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:40.036 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.059+0000 7feb837fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:40.369 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.392+0000 7feba099c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:40.369 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.392+0000 7feba099c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:40.369 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.392+0000 7feba099c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:22:40.369 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.392+0000 7feba099c640 1 -- 172.21.15.17:0/3867930512 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb9402f5e0 con 0x7feb94030660 2024-03-29T11:22:40.375 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:22:40.375 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:40.376 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.396+0000 7feb837fe640 1 -- 172.21.15.17:0/3867930512 <== mon.1 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7feb900525e0 con 0x7feb94030660 2024-03-29T11:22:40.376 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.396+0000 7feb837fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:40.376 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.396+0000 7feb837fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:40.377 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.399+0000 7feba099c640 10 librados: watch_flush enter 2024-03-29T11:22:40.377 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.400+0000 7feba099c640 10 librados: watch_flush exit 2024-03-29T11:22:40.377 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.400+0000 7feba099c640 20 client.4369.objecter shutdown clearing up homeless session... 2024-03-29T11:22:40.377 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.400+0000 7feba099c640 10 client.4369.objecter successfully canceled tick 2024-03-29T11:22:40.377 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.400+0000 7feba099c640 1 -- 172.21.15.17:0/3867930512 >> v1:172.21.15.17:6816/3236185052 conn(0x7feb8806dca0 legacy=0x7feb88070160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:40.377 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.400+0000 7feba099c640 10 monclient: shutdown 2024-03-29T11:22:40.377 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.400+0000 7feba099c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:40.377 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.400+0000 7feba099c640 1 -- 172.21.15.17:0/3867930512 >> v1:172.21.15.94:6789/0 conn(0x7feb94030660 legacy=0x7feb9415fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:40.377 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.400+0000 7feba099c640 1 -- 172.21.15.17:0/3867930512 shutdown_connections 2024-03-29T11:22:40.378 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.400+0000 7feba099c640 1 -- 172.21.15.17:0/3867930512 >> 172.21.15.17:0/3867930512 conn(0x7feb94005c00 msgr2=0x7feb94034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:40.380 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.403+0000 7feba099c640 1 -- 172.21.15.17:0/3867930512 shutdown_connections 2024-03-29T11:22:40.380 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.403+0000 7feba099c640 1 -- 172.21.15.17:0/3867930512 wait complete. 2024-03-29T11:22:40.380 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:40.403+0000 7feba099c640 1 librados: shutdown 2024-03-29T11:22:40.397 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:47.203 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:47.353 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.376+0000 7f847473e640 10 monclient: get_monmap_and_config 2024-03-29T11:22:47.353 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.376+0000 7f847473e640 10 monclient: build_initial_monmap 2024-03-29T11:22:47.353 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.376+0000 7f847473e640 10 monclient: monmap: 2024-03-29T11:22:47.353 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:22:47.353 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:22:47.354 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:22:47.377115+0000 2024-03-29T11:22:47.354 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:22:47.377115+0000 2024-03-29T11:22:47.354 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:47.354 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:47.354 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:22:47.354 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:22:47.354 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:22:47.354 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:47.356 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.379+0000 7f847473e640 1 Processor -- start 2024-03-29T11:22:47.357 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.379+0000 7f847473e640 1 -- start start 2024-03-29T11:22:47.357 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.379+0000 7f847473e640 10 monclient: init 2024-03-29T11:22:47.357 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.380+0000 7f847473e640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:47.358 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.381+0000 7f847473e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:22:47.360 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.381+0000 7f847473e640 10 monclient(hunting): picked mon.noname-c con 0x7f846c0b0770 addr v1:172.21.15.94:6789/0 2024-03-29T11:22:47.360 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.381+0000 7f847473e640 10 monclient(hunting): picked mon.noname-b con 0x7f846c14a700 addr v1:172.21.15.17:6790/0 2024-03-29T11:22:47.360 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.382+0000 7f847473e640 10 monclient(hunting): picked mon.noname-a con 0x7f846c0ac5a0 addr v1:172.21.15.17:6789/0 2024-03-29T11:22:47.360 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.382+0000 7f847473e640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f846c0aba70 con 0x7f846c0ac5a0 2024-03-29T11:22:47.360 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.382+0000 7f847473e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f846c0aecb0 con 0x7f846c0b0770 2024-03-29T11:22:47.360 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.382+0000 7f847473e640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f846c0af150 con 0x7f846c14a700 2024-03-29T11:22:47.360 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.382+0000 7f847473e640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:47.360 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.382+0000 7f847473e640 10 monclient(hunting): authenticate will time out at 1073.587114s 2024-03-29T11:22:47.361 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.384+0000 7f84724da640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f846c0b0770 0x7f846c149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:48563/0 (socket says 172.21.15.17:48563) 2024-03-29T11:22:47.361 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.384+0000 7f84724da640 1 -- 172.21.15.17:0/1827017385 learned_addr learned my addr 172.21.15.17:0/1827017385 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:22:47.363 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.386+0000 7f84714d8640 1 -- 172.21.15.17:0/1827017385 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1387892173 0 0) 0x7f846c0aba70 con 0x7f846c0ac5a0 2024-03-29T11:22:47.364 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.386+0000 7f84714d8640 10 monclient(hunting): _negotiate 2024-03-29T11:22:47.365 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.386+0000 7f84714d8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:47.365 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.386+0000 7f84714d8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:47.365 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.386+0000 7f84714d8640 10 monclient(hunting): my global_id is 4365 2024-03-29T11:22:47.365 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.386+0000 7f84714d8640 1 -- 172.21.15.17:0/1827017385 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8458003520 con 0x7f846c0ac5a0 2024-03-29T11:22:47.365 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.388+0000 7f84714d8640 1 -- 172.21.15.17:0/1827017385 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2437023999 0 0) 0x7f846c0aecb0 con 0x7f846c0b0770 2024-03-29T11:22:47.365 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.388+0000 7f84714d8640 10 monclient(hunting): _negotiate 2024-03-29T11:22:47.365 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.388+0000 7f84714d8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:47.365 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.388+0000 7f84714d8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:47.365 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.388+0000 7f84714d8640 10 monclient(hunting): my global_id is 4372 2024-03-29T11:22:47.366 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.388+0000 7f84714d8640 1 -- 172.21.15.17:0/1827017385 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f846c0aba70 con 0x7f846c0b0770 2024-03-29T11:22:47.366 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.388+0000 7f84714d8640 1 -- 172.21.15.17:0/1827017385 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3908844210 0 0) 0x7f8458003520 con 0x7f846c0ac5a0 2024-03-29T11:22:47.366 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.388+0000 7f84714d8640 1 -- 172.21.15.17:0/1827017385 >> v1:172.21.15.17:6790/0 conn(0x7f846c14a700 legacy=0x7f846c0abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:47.369 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.389+0000 7f84714d8640 1 -- 172.21.15.17:0/1827017385 >> v1:172.21.15.94:6789/0 conn(0x7f846c0b0770 legacy=0x7f846c149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:47.369 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.389+0000 7f84714d8640 10 monclient: _finish_hunting 0 2024-03-29T11:22:47.369 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.389+0000 7f84714d8640 1 monclient: found mon.noname-a 2024-03-29T11:22:47.369 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.389+0000 7f84714d8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:47.369 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.389+0000 7f84714d8640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.17:6789/0 2024-03-29T11:22:47.369 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.389+0000 7f84714d8640 1 -- 172.21.15.17:0/1827017385 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f846c0ac9d0 con 0x7f846c0ac5a0 2024-03-29T11:22:47.369 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.389+0000 7f84714d8640 10 monclient: _finish_auth 0 2024-03-29T11:22:47.369 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.389+0000 7f84714d8640 10 monclient: _check_auth_tickets 2024-03-29T11:22:47.369 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.389+0000 7f84714d8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:47.369 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.389+0000 7f84714d8640 1 -- 172.21.15.17:0/1827017385 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f8468001610 con 0x7f846c0ac5a0 2024-03-29T11:22:47.369 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.389+0000 7f84714d8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:47.369 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.389+0000 7f84714d8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:47.369 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.389+0000 7f84714d8640 10 monclient: dump: 2024-03-29T11:22:47.369 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:22:47.370 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:22:47.370 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:47.370 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:47.370 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:47.370 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:47.370 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:22:47.370 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:22:47.370 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:22:47.370 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:47.370 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.389+0000 7f847473e640 5 monclient: authenticate success, global_id 4365 2024-03-29T11:22:47.370 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.389+0000 7f847473e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:47.370 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.389+0000 7f84714d8640 1 -- 172.21.15.17:0/1827017385 <== mon.0 v1:172.21.15.17:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8468001770 con 0x7f846c0ac5a0 2024-03-29T11:22:47.370 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.389+0000 7f84714d8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:47.370 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.389+0000 7f84714d8640 1 -- 172.21.15.17:0/1827017385 <== mon.0 v1:172.21.15.17:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f8468001bc0 con 0x7f846c0ac5a0 2024-03-29T11:22:47.371 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.389+0000 7f847473e640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:47.371 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.389+0000 7f84714d8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:47.371 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.389+0000 7f84714d8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:47.371 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.389+0000 7f84714d8640 10 monclient: dump: 2024-03-29T11:22:47.371 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:22:47.371 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:22:47.371 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:47.371 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:47.371 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:47.371 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:47.371 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:22:47.371 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:22:47.371 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:22:47.371 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:47.372 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.389+0000 7f847473e640 4 set_mon_vals no callback set 2024-03-29T11:22:47.372 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.390+0000 7f847473e640 10 monclient: shutdown 2024-03-29T11:22:47.372 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.390+0000 7f847473e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:47.372 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.390+0000 7f847473e640 1 -- 172.21.15.17:0/1827017385 >> v1:172.21.15.17:6789/0 conn(0x7f846c0ac5a0 legacy=0x7f846c151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:47.372 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.391+0000 7f847473e640 1 -- 172.21.15.17:0/1827017385 shutdown_connections 2024-03-29T11:22:47.372 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.391+0000 7f847473e640 1 -- 172.21.15.17:0/1827017385 >> 172.21.15.17:0/1827017385 conn(0x7f846c06b660 msgr2=0x7f846c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:47.372 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.393+0000 7f847473e640 1 -- 172.21.15.17:0/1827017385 shutdown_connections 2024-03-29T11:22:47.373 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.393+0000 7f847473e640 1 -- 172.21.15.17:0/1827017385 wait complete. 2024-03-29T11:22:47.373 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.394+0000 7f847473e640 10 monclient: build_initial_monmap 2024-03-29T11:22:47.373 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.394+0000 7f847473e640 10 monclient: monmap: 2024-03-29T11:22:47.373 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:22:47.373 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:47.373 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:22:47.373 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:22:47.373 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:47.373 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:47.373 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:22:47.373 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:22:47.373 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:22:47.373 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:47.373 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.395+0000 7f847473e640 1 librados: starting msgr at 2024-03-29T11:22:47.373 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.395+0000 7f847473e640 1 librados: starting objecter 2024-03-29T11:22:47.374 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.395+0000 7f847473e640 1 Processor -- start 2024-03-29T11:22:47.376 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.396+0000 7f847473e640 1 -- start start 2024-03-29T11:22:47.376 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.396+0000 7f847473e640 1 librados: setting wanted keys 2024-03-29T11:22:47.377 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.396+0000 7f847473e640 1 librados: calling monclient init 2024-03-29T11:22:47.377 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.396+0000 7f847473e640 10 monclient: init 2024-03-29T11:22:47.377 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.397+0000 7f847473e640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:47.377 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.397+0000 7f847473e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:47.382 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.400+0000 7f847473e640 10 monclient(hunting): picked mon.noname-c con 0x7f846c0ac5a0 addr v1:172.21.15.17:6790/0 2024-03-29T11:22:47.382 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.400+0000 7f847473e640 10 monclient(hunting): picked mon.noname-b con 0x7f846c0b0770 addr v1:172.21.15.94:6789/0 2024-03-29T11:22:47.382 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.400+0000 7f847473e640 10 monclient(hunting): picked mon.noname-a con 0x7f846c14a700 addr v1:172.21.15.17:6789/0 2024-03-29T11:22:47.382 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.400+0000 7f847473e640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f846c1283a0 con 0x7f846c14a700 2024-03-29T11:22:47.382 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.400+0000 7f847473e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f846c1285d0 con 0x7f846c0b0770 2024-03-29T11:22:47.383 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.400+0000 7f847473e640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f846c145050 con 0x7f846c0ac5a0 2024-03-29T11:22:47.383 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.400+0000 7f847473e640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:47.383 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.400+0000 7f847473e640 10 monclient(hunting): authenticate will time out at 1073.605280s 2024-03-29T11:22:47.383 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.403+0000 7f84724da640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f846c0ac5a0 0x7f846c1df170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:41257/0 (socket says 172.21.15.17:41257) 2024-03-29T11:22:47.383 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.403+0000 7f84724da640 1 -- 172.21.15.17:0/970048986 learned_addr learned my addr 172.21.15.17:0/970048986 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:22:47.383 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.404+0000 7f8462ffd640 10 client.?.objecter ms_handle_connect 0x7f846c0ac5a0 2024-03-29T11:22:47.383 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.404+0000 7f8462ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:47.383 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.404+0000 7f8462ffd640 1 -- 172.21.15.17:0/970048986 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1564647317 0 0) 0x7f846c145050 con 0x7f846c0ac5a0 2024-03-29T11:22:47.383 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.404+0000 7f8462ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:47.383 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.404+0000 7f8462ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:47.383 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.404+0000 7f8462ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:47.383 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.404+0000 7f8462ffd640 10 monclient(hunting): my global_id is 4391 2024-03-29T11:22:47.383 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.404+0000 7f8462ffd640 1 -- 172.21.15.17:0/970048986 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8450004eb0 con 0x7f846c0ac5a0 2024-03-29T11:22:47.383 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.404+0000 7f8462ffd640 10 client.?.objecter ms_handle_connect 0x7f846c0b0770 2024-03-29T11:22:47.384 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.404+0000 7f8462ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:47.384 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.404+0000 7f8462ffd640 1 -- 172.21.15.17:0/970048986 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2917159047 0 0) 0x7f8450004eb0 con 0x7f846c0ac5a0 2024-03-29T11:22:47.384 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.404+0000 7f8462ffd640 1 -- 172.21.15.17:0/970048986 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f846c145050 con 0x7f846c0ac5a0 2024-03-29T11:22:47.384 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.404+0000 7f8462ffd640 1 -- 172.21.15.17:0/970048986 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f845c001310 con 0x7f846c0ac5a0 2024-03-29T11:22:47.384 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.404+0000 7f8462ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:47.384 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.404+0000 7f8462ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:47.384 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.404+0000 7f8462ffd640 10 monclient(hunting): dump: 2024-03-29T11:22:47.384 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:22:47.384 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:22:47.384 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:47.384 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:47.384 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:47.384 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:47.384 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:22:47.385 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:22:47.385 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:22:47.385 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:47.385 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.404+0000 7f8462ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:47.387 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.408+0000 7f8462ffd640 1 -- 172.21.15.17:0/970048986 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1335484198 0 0) 0x7f846c1285d0 con 0x7f846c0b0770 2024-03-29T11:22:47.387 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.408+0000 7f8462ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:47.387 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.408+0000 7f8462ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:47.388 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.408+0000 7f8462ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:47.388 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.408+0000 7f8462ffd640 10 monclient(hunting): my global_id is 4375 2024-03-29T11:22:47.388 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.408+0000 7f8462ffd640 1 -- 172.21.15.17:0/970048986 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8450004eb0 con 0x7f846c0b0770 2024-03-29T11:22:47.388 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.408+0000 7f8462ffd640 1 -- 172.21.15.17:0/970048986 <== mon.2 v1:172.21.15.17:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3663443732 0 0) 0x7f846c145050 con 0x7f846c0ac5a0 2024-03-29T11:22:47.388 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.408+0000 7f8462ffd640 1 -- 172.21.15.17:0/970048986 >> v1:172.21.15.94:6789/0 conn(0x7f846c0b0770 legacy=0x7f846c1443d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:47.388 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.408+0000 7f8462ffd640 1 -- 172.21.15.17:0/970048986 >> v1:172.21.15.17:6789/0 conn(0x7f846c14a700 legacy=0x7f846c126250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:47.388 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.408+0000 7f8462ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:22:47.388 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.408+0000 7f8462ffd640 1 monclient: found mon.c 2024-03-29T11:22:47.388 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.408+0000 7f8462ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:47.388 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.408+0000 7f8462ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:22:47.388 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.408+0000 7f8462ffd640 1 -- 172.21.15.17:0/970048986 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f846c1451e0 con 0x7f846c0ac5a0 2024-03-29T11:22:47.388 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.408+0000 7f8462ffd640 10 monclient: _finish_auth 0 2024-03-29T11:22:47.388 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.408+0000 7f8462ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:22:47.388 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.408+0000 7f8462ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:47.388 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.409+0000 7f847473e640 5 monclient: authenticate success, global_id 4391 2024-03-29T11:22:47.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.409+0000 7f847473e640 10 monclient: _renew_subs 2024-03-29T11:22:47.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.409+0000 7f847473e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:22:47.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.409+0000 7f847473e640 1 -- 172.21.15.17:0/970048986 --> v1:172.21.15.17:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f846c144ae0 con 0x7f846c0ac5a0 2024-03-29T11:22:47.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.409+0000 7f847473e640 10 client.4391.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:47.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.409+0000 7f847473e640 10 monclient: _renew_subs 2024-03-29T11:22:47.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.409+0000 7f847473e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:22:47.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.409+0000 7f847473e640 1 -- 172.21.15.17:0/970048986 --> v1:172.21.15.17:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f846c126b40 con 0x7f846c0ac5a0 2024-03-29T11:22:47.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.409+0000 7f847473e640 1 librados: init done 2024-03-29T11:22:47.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.411+0000 7f847473e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:47.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.411+0000 7f847473e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:47.390 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.411+0000 7f847473e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:22:47.390 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.411+0000 7f847473e640 1 -- 172.21.15.17:0/970048986 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f846c144e80 con 0x7f846c0ac5a0 2024-03-29T11:22:47.391 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.414+0000 7f8462ffd640 1 -- 172.21.15.17:0/970048986 <== mon.2 v1:172.21.15.17:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f845c0047e0 con 0x7f846c0ac5a0 2024-03-29T11:22:47.391 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.414+0000 7f8462ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:47.391 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.414+0000 7f8462ffd640 1 -- 172.21.15.17:0/970048986 <== mon.2 v1:172.21.15.17:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f845c004af0 con 0x7f846c0ac5a0 2024-03-29T11:22:47.391 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.414+0000 7f8462ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:47.392 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.414+0000 7f8462ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:47.392 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.414+0000 7f8462ffd640 10 monclient: dump: 2024-03-29T11:22:47.392 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:22:47.392 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:22:47.392 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:47.392 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:47.392 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:47.392 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:47.392 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:22:47.392 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:22:47.392 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:22:47.392 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:47.392 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.414+0000 7f8470cd7640 4 set_mon_vals no callback set 2024-03-29T11:22:47.394 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.416+0000 7f8462ffd640 1 -- 172.21.15.17:0/970048986 <== mon.2 v1:172.21.15.17:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f845c01ab00 con 0x7f846c0ac5a0 2024-03-29T11:22:47.394 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.417+0000 7f8462ffd640 1 -- 172.21.15.17:0/970048986 <== mon.2 v1:172.21.15.17:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f845c084ae0 con 0x7f846c0ac5a0 2024-03-29T11:22:47.395 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.417+0000 7f8462ffd640 10 client.4391.objecter ms_dispatch 0x7f846c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:47.395 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.417+0000 7f8462ffd640 3 client.4391.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:47.395 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.417+0000 7f8462ffd640 3 client.4391.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:47.395 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.417+0000 7f8462ffd640 20 client.4391.objecter dump_active .. 0 homeless 2024-03-29T11:22:47.395 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.417+0000 7f8471cd9640 10 monclient: get_auth_request con 0x7f845006dce0 auth_method 0 2024-03-29T11:22:47.395 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.417+0000 7f8462ffd640 10 client.4391.objecter ms_handle_connect 0x7f845006dce0 2024-03-29T11:22:47.406 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.428+0000 7f8462ffd640 1 -- 172.21.15.17:0/970048986 <== mon.2 v1:172.21.15.17:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f845c0503c0 con 0x7f846c0ac5a0 2024-03-29T11:22:47.406 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.428+0000 7f8462ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:47.406 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.428+0000 7f8462ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:47.733 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.756+0000 7f847473e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:47.733 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.756+0000 7f847473e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:47.733 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.756+0000 7f847473e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:22:47.734 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.756+0000 7f847473e640 1 -- 172.21.15.17:0/970048986 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f846c1df880 con 0x7f846c0ac5a0 2024-03-29T11:22:47.734 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.756+0000 7f8462ffd640 1 -- 172.21.15.17:0/970048986 <== mon.2 v1:172.21.15.17:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f845c0522f0 con 0x7f846c0ac5a0 2024-03-29T11:22:47.734 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.756+0000 7f8462ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:47.734 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.756+0000 7f8462ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:47.735 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:22:47.735 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:47.739 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.760+0000 7f847473e640 10 librados: watch_flush enter 2024-03-29T11:22:47.739 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.760+0000 7f847473e640 10 librados: watch_flush exit 2024-03-29T11:22:47.739 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.761+0000 7f847473e640 20 client.4391.objecter shutdown clearing up homeless session... 2024-03-29T11:22:47.739 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.761+0000 7f847473e640 10 client.4391.objecter successfully canceled tick 2024-03-29T11:22:47.740 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.761+0000 7f847473e640 1 -- 172.21.15.17:0/970048986 >> v1:172.21.15.17:6816/3236185052 conn(0x7f845006dce0 legacy=0x7f84500701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:47.740 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.761+0000 7f847473e640 10 monclient: shutdown 2024-03-29T11:22:47.740 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.761+0000 7f847473e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:47.740 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.761+0000 7f847473e640 1 -- 172.21.15.17:0/970048986 >> v1:172.21.15.17:6790/0 conn(0x7f846c0ac5a0 legacy=0x7f846c1df170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:47.740 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.761+0000 7f847473e640 1 -- 172.21.15.17:0/970048986 shutdown_connections 2024-03-29T11:22:47.740 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.761+0000 7f847473e640 1 -- 172.21.15.17:0/970048986 >> 172.21.15.17:0/970048986 conn(0x7f846c06b660 msgr2=0x7f846c0b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:47.740 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.761+0000 7f847473e640 1 -- 172.21.15.17:0/970048986 shutdown_connections 2024-03-29T11:22:47.740 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.761+0000 7f847473e640 1 -- 172.21.15.17:0/970048986 wait complete. 2024-03-29T11:22:47.740 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:47.761+0000 7f847473e640 1 librados: shutdown 2024-03-29T11:22:47.755 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:54.561 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:54.713 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.734+0000 7efeda36b640 10 monclient: get_monmap_and_config 2024-03-29T11:22:54.713 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.734+0000 7efeda36b640 10 monclient: build_initial_monmap 2024-03-29T11:22:54.713 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.734+0000 7efeda36b640 10 monclient: monmap: 2024-03-29T11:22:54.713 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:22:54.713 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:22:54.713 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:22:54.735430+0000 2024-03-29T11:22:54.713 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:22:54.735430+0000 2024-03-29T11:22:54.713 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:54.713 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:54.714 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:22:54.714 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:22:54.714 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:22:54.714 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:54.715 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.736+0000 7efeda36b640 1 Processor -- start 2024-03-29T11:22:54.715 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.736+0000 7efeda36b640 1 -- start start 2024-03-29T11:22:54.715 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.736+0000 7efeda36b640 10 monclient: init 2024-03-29T11:22:54.716 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.737+0000 7efeda36b640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:54.716 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.737+0000 7efeda36b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:54.716 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.737+0000 7efeda36b640 10 monclient(hunting): picked mon.noname-b con 0x7efed40b4100 addr v1:172.21.15.17:6790/0 2024-03-29T11:22:54.716 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.737+0000 7efeda36b640 10 monclient(hunting): picked mon.noname-c con 0x7efed413ee90 addr v1:172.21.15.94:6789/0 2024-03-29T11:22:54.716 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.737+0000 7efeda36b640 10 monclient(hunting): picked mon.noname-a con 0x7efed40af900 addr v1:172.21.15.17:6789/0 2024-03-29T11:22:54.716 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.737+0000 7efeda36b640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed40afd10 con 0x7efed40af900 2024-03-29T11:22:54.716 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.737+0000 7efeda36b640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed40af6f0 con 0x7efed413ee90 2024-03-29T11:22:54.716 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.737+0000 7efeda36b640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed40aaf60 con 0x7efed40b4100 2024-03-29T11:22:54.716 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.737+0000 7efeda36b640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:54.716 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.737+0000 7efeda36b640 10 monclient(hunting): authenticate will time out at 1080.942764s 2024-03-29T11:22:54.718 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.741+0000 7efed37fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7efed413ee90 0x7efed40adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:36675/0 (socket says 172.21.15.17:36675) 2024-03-29T11:22:54.718 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.741+0000 7efed37fe640 1 -- 172.21.15.17:0/2953541377 learned_addr learned my addr 172.21.15.17:0/2953541377 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:22:54.719 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.741+0000 7efed2ffd640 1 -- 172.21.15.17:0/2953541377 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3589721416 0 0) 0x7efed40af6f0 con 0x7efed413ee90 2024-03-29T11:22:54.719 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.741+0000 7efed2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:54.719 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.741+0000 7efed2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:54.719 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.741+0000 7efed2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:54.719 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.742+0000 7efed2ffd640 10 monclient(hunting): my global_id is 4378 2024-03-29T11:22:54.719 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.742+0000 7efed2ffd640 1 -- 172.21.15.17:0/2953541377 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efebc003520 con 0x7efed413ee90 2024-03-29T11:22:54.720 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.743+0000 7efed2ffd640 1 -- 172.21.15.17:0/2953541377 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2412467502 0 0) 0x7efebc003520 con 0x7efed413ee90 2024-03-29T11:22:54.721 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.743+0000 7efed2ffd640 1 -- 172.21.15.17:0/2953541377 >> v1:172.21.15.17:6790/0 conn(0x7efed40b4100 legacy=0x7efed40a9850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:54.721 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.743+0000 7efed2ffd640 1 -- 172.21.15.17:0/2953541377 >> v1:172.21.15.17:6789/0 conn(0x7efed40af900 legacy=0x7efed413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:54.721 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.743+0000 7efed2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:22:54.721 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.743+0000 7efed2ffd640 1 monclient: found mon.noname-c 2024-03-29T11:22:54.721 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.743+0000 7efed2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:54.721 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.743+0000 7efed2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6789/0 2024-03-29T11:22:54.721 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.743+0000 7efed2ffd640 1 -- 172.21.15.17:0/2953541377 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed413e970 con 0x7efed413ee90 2024-03-29T11:22:54.721 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.743+0000 7efed2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:22:54.721 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.743+0000 7efed2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:22:54.721 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.743+0000 7efed2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:54.721 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.743+0000 7efed2ffd640 1 -- 172.21.15.17:0/2953541377 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7efecc001a10 con 0x7efed413ee90 2024-03-29T11:22:54.721 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.743+0000 7efed2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:54.721 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.743+0000 7efed2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:54.721 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.743+0000 7efed2ffd640 10 monclient: dump: 2024-03-29T11:22:54.722 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:22:54.722 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:22:54.722 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:54.722 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:54.722 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:54.722 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:54.722 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:22:54.722 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:22:54.722 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:22:54.722 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:54.722 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.743+0000 7efeda36b640 5 monclient: authenticate success, global_id 4378 2024-03-29T11:22:54.722 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.743+0000 7efeda36b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:54.723 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.744+0000 7efed2ffd640 1 -- 172.21.15.17:0/2953541377 <== mon.1 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efecc001be0 con 0x7efed413ee90 2024-03-29T11:22:54.723 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.744+0000 7efed2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:54.723 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.744+0000 7efeda36b640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:54.723 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.744+0000 7efeda36b640 4 set_mon_vals no callback set 2024-03-29T11:22:54.724 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.745+0000 7efeda36b640 10 monclient: shutdown 2024-03-29T11:22:54.724 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.745+0000 7efeda36b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:54.724 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.745+0000 7efeda36b640 1 -- 172.21.15.17:0/2953541377 >> v1:172.21.15.94:6789/0 conn(0x7efed413ee90 legacy=0x7efed40adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:54.726 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.748+0000 7efeda36b640 1 -- 172.21.15.17:0/2953541377 shutdown_connections 2024-03-29T11:22:54.726 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.748+0000 7efeda36b640 1 -- 172.21.15.17:0/2953541377 >> 172.21.15.17:0/2953541377 conn(0x7efed40b35e0 msgr2=0x7efed40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:54.728 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.750+0000 7efeda36b640 1 -- 172.21.15.17:0/2953541377 shutdown_connections 2024-03-29T11:22:54.732 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.752+0000 7efeda36b640 1 -- 172.21.15.17:0/2953541377 wait complete. 2024-03-29T11:22:54.732 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.752+0000 7efeda36b640 10 monclient: build_initial_monmap 2024-03-29T11:22:54.733 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.752+0000 7efeda36b640 10 monclient: monmap: 2024-03-29T11:22:54.733 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:22:54.733 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:54.733 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:22:54.733 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:22:54.733 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:54.733 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:54.733 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:22:54.733 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:22:54.733 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:22:54.733 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:54.733 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.753+0000 7efeda36b640 1 librados: starting msgr at 2024-03-29T11:22:54.733 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.753+0000 7efeda36b640 1 librados: starting objecter 2024-03-29T11:22:54.733 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.753+0000 7efeda36b640 1 Processor -- start 2024-03-29T11:22:54.737 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.754+0000 7efeda36b640 1 -- start start 2024-03-29T11:22:54.737 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.754+0000 7efeda36b640 1 librados: setting wanted keys 2024-03-29T11:22:54.737 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.754+0000 7efeda36b640 1 librados: calling monclient init 2024-03-29T11:22:54.737 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.754+0000 7efeda36b640 10 monclient: init 2024-03-29T11:22:54.737 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.755+0000 7efeda36b640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:54.737 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.755+0000 7efeda36b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:22:54.737 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.755+0000 7efeda36b640 10 monclient(hunting): picked mon.noname-c con 0x7efed40af900 addr v1:172.21.15.17:6790/0 2024-03-29T11:22:54.738 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.755+0000 7efeda36b640 10 monclient(hunting): picked mon.noname-a con 0x7efed40b4100 addr v1:172.21.15.17:6789/0 2024-03-29T11:22:54.738 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.755+0000 7efeda36b640 10 monclient(hunting): picked mon.noname-b con 0x7efed413ee90 addr v1:172.21.15.94:6789/0 2024-03-29T11:22:54.738 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.755+0000 7efeda36b640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed414fa00 con 0x7efed40b4100 2024-03-29T11:22:54.738 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.755+0000 7efeda36b640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed41516b0 con 0x7efed413ee90 2024-03-29T11:22:54.738 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.755+0000 7efeda36b640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed4151990 con 0x7efed40af900 2024-03-29T11:22:54.738 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.755+0000 7efeda36b640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:54.738 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.755+0000 7efeda36b640 10 monclient(hunting): authenticate will time out at 1080.960417s 2024-03-29T11:22:54.739 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.760+0000 7efed8908640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7efed413ee90 0x7efed414e540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:36767/0 (socket says 172.21.15.17:36767) 2024-03-29T11:22:54.739 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.760+0000 7efed8908640 1 -- 172.21.15.17:0/2686105132 learned_addr learned my addr 172.21.15.17:0/2686105132 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:22:54.739 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.761+0000 7efed0ff9640 10 client.?.objecter ms_handle_connect 0x7efed413ee90 2024-03-29T11:22:54.739 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.761+0000 7efed0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:54.740 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.762+0000 7efed0ff9640 10 client.?.objecter ms_handle_connect 0x7efed40af900 2024-03-29T11:22:54.740 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.762+0000 7efed0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:54.740 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.763+0000 7efed0ff9640 10 client.?.objecter ms_handle_connect 0x7efed40b4100 2024-03-29T11:22:54.740 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.763+0000 7efed0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:54.743 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.763+0000 7efed0ff9640 1 -- 172.21.15.17:0/2686105132 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 422624708 0 0) 0x7efed414fa00 con 0x7efed40b4100 2024-03-29T11:22:54.743 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.763+0000 7efed0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:22:54.743 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.763+0000 7efed0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:54.743 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.763+0000 7efed0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:54.743 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.763+0000 7efed0ff9640 10 monclient(hunting): my global_id is 4368 2024-03-29T11:22:54.743 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.763+0000 7efed0ff9640 1 -- 172.21.15.17:0/2686105132 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efec0004eb0 con 0x7efed40b4100 2024-03-29T11:22:54.743 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.763+0000 7efed0ff9640 1 -- 172.21.15.17:0/2686105132 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2975501980 0 0) 0x7efed4151990 con 0x7efed40af900 2024-03-29T11:22:54.743 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.763+0000 7efed0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:22:54.743 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.763+0000 7efed0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:54.743 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.763+0000 7efed0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:54.743 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.763+0000 7efed0ff9640 10 monclient(hunting): my global_id is 4394 2024-03-29T11:22:54.743 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.763+0000 7efed0ff9640 1 -- 172.21.15.17:0/2686105132 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efed414fa00 con 0x7efed40af900 2024-03-29T11:22:54.743 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.763+0000 7efed0ff9640 1 -- 172.21.15.17:0/2686105132 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2855318005 0 0) 0x7efec0004eb0 con 0x7efed40b4100 2024-03-29T11:22:54.744 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.763+0000 7efed0ff9640 1 -- 172.21.15.17:0/2686105132 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efed4151990 con 0x7efed40b4100 2024-03-29T11:22:54.744 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.763+0000 7efed0ff9640 1 -- 172.21.15.17:0/2686105132 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7efecc004b60 con 0x7efed40b4100 2024-03-29T11:22:54.744 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.764+0000 7efed0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:54.744 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.764+0000 7efed0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:54.744 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.764+0000 7efed0ff9640 10 monclient(hunting): dump: 2024-03-29T11:22:54.744 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:22:54.744 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:22:54.744 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:54.744 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:54.744 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:54.744 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:54.744 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:22:54.744 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:22:54.744 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:22:54.744 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:54.745 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.764+0000 7efed0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:54.745 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.764+0000 7efed0ff9640 1 -- 172.21.15.17:0/2686105132 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1268131937 0 0) 0x7efed414fa00 con 0x7efed40af900 2024-03-29T11:22:54.745 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.764+0000 7efed0ff9640 1 -- 172.21.15.17:0/2686105132 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efec0004eb0 con 0x7efed40af900 2024-03-29T11:22:54.745 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.764+0000 7efed0ff9640 1 -- 172.21.15.17:0/2686105132 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7efec8001790 con 0x7efed40af900 2024-03-29T11:22:54.745 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.764+0000 7efed0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:54.745 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.764+0000 7efed0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:54.745 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.764+0000 7efed0ff9640 10 monclient(hunting): dump: 2024-03-29T11:22:54.745 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:22:54.745 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:22:54.745 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:54.745 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:54.745 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:54.745 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:54.745 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:22:54.746 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:22:54.746 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:22:54.746 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:54.746 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.765+0000 7efed0ff9640 1 -- 172.21.15.17:0/2686105132 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 118836738 0 0) 0x7efed41516b0 con 0x7efed413ee90 2024-03-29T11:22:54.746 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.765+0000 7efed0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:22:54.746 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.765+0000 7efed0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:54.746 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.765+0000 7efed0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:54.746 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.765+0000 7efed0ff9640 10 monclient(hunting): my global_id is 4381 2024-03-29T11:22:54.746 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.765+0000 7efed0ff9640 1 -- 172.21.15.17:0/2686105132 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efed414fa00 con 0x7efed413ee90 2024-03-29T11:22:54.750 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.766+0000 7efed0ff9640 1 -- 172.21.15.17:0/2686105132 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2222676742 0 0) 0x7efed414fa00 con 0x7efed413ee90 2024-03-29T11:22:54.750 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.766+0000 7efed0ff9640 1 -- 172.21.15.17:0/2686105132 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efed41516b0 con 0x7efed413ee90 2024-03-29T11:22:54.750 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.766+0000 7efed0ff9640 1 -- 172.21.15.17:0/2686105132 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7efec4001820 con 0x7efed413ee90 2024-03-29T11:22:54.750 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.766+0000 7efed0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:54.750 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.766+0000 7efed0ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:54.750 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.766+0000 7efed0ff9640 10 monclient(hunting): dump: 2024-03-29T11:22:54.750 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:22:54.750 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:22:54.750 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:54.750 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:54.750 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:54.751 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:54.751 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:22:54.751 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:22:54.751 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:22:54.751 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:54.751 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.767+0000 7efed0ff9640 1 -- 172.21.15.17:0/2686105132 <== mon.2 v1:172.21.15.17:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1683841255 0 0) 0x7efec0004eb0 con 0x7efed40af900 2024-03-29T11:22:54.751 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.767+0000 7efed0ff9640 1 -- 172.21.15.17:0/2686105132 >> v1:172.21.15.94:6789/0 conn(0x7efed413ee90 legacy=0x7efed414e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:54.751 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.767+0000 7efed0ff9640 1 -- 172.21.15.17:0/2686105132 >> v1:172.21.15.17:6789/0 conn(0x7efed40b4100 legacy=0x7efed414de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:54.751 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.767+0000 7efed0ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:22:54.751 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.767+0000 7efed0ff9640 1 monclient: found mon.c 2024-03-29T11:22:54.751 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.767+0000 7efed0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:54.751 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.767+0000 7efed0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:22:54.751 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.767+0000 7efed0ff9640 1 -- 172.21.15.17:0/2686105132 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed412a4c0 con 0x7efed40af900 2024-03-29T11:22:54.751 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.767+0000 7efed0ff9640 10 monclient: _finish_auth 0 2024-03-29T11:22:54.751 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.767+0000 7efed0ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:22:54.752 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.767+0000 7efed0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:54.752 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.767+0000 7efeda36b640 5 monclient: authenticate success, global_id 4394 2024-03-29T11:22:54.752 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.767+0000 7efeda36b640 10 monclient: _renew_subs 2024-03-29T11:22:54.752 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.767+0000 7efeda36b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:22:54.752 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.767+0000 7efeda36b640 1 -- 172.21.15.17:0/2686105132 --> v1:172.21.15.17:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efed414ec50 con 0x7efed40af900 2024-03-29T11:22:54.752 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.767+0000 7efeda36b640 10 client.4394.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:54.752 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.767+0000 7efeda36b640 10 monclient: _renew_subs 2024-03-29T11:22:54.752 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.767+0000 7efeda36b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:22:54.752 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.767+0000 7efeda36b640 1 -- 172.21.15.17:0/2686105132 --> v1:172.21.15.17:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efed4151f90 con 0x7efed40af900 2024-03-29T11:22:54.752 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.767+0000 7efeda36b640 1 librados: init done 2024-03-29T11:22:54.752 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.767+0000 7efed0ff9640 1 -- 172.21.15.17:0/2686105132 <== mon.2 v1:172.21.15.17:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efec8001c80 con 0x7efed40af900 2024-03-29T11:22:54.752 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.767+0000 7efed0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:54.752 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.767+0000 7efed0ff9640 1 -- 172.21.15.17:0/2686105132 <== mon.2 v1:172.21.15.17:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7efec8003790 con 0x7efed40af900 2024-03-29T11:22:54.752 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.767+0000 7efed0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:54.752 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.767+0000 7efed0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:54.753 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.767+0000 7efed0ff9640 10 monclient: dump: 2024-03-29T11:22:54.753 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:22:54.753 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:22:54.753 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:54.753 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:22:54.753 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:54.753 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:22:54.753 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:22:54.753 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:22:54.753 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:22:54.753 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:22:54.753 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.768+0000 7efed27fc640 4 set_mon_vals no callback set 2024-03-29T11:22:54.753 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.770+0000 7efeda36b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:54.753 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.770+0000 7efeda36b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:54.754 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.770+0000 7efeda36b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:22:54.754 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.770+0000 7efeda36b640 1 -- 172.21.15.17:0/2686105132 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efeb4005080 con 0x7efed40af900 2024-03-29T11:22:54.754 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.771+0000 7efed0ff9640 1 -- 172.21.15.17:0/2686105132 <== mon.2 v1:172.21.15.17:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7efec80038f0 con 0x7efed40af900 2024-03-29T11:22:54.754 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.771+0000 7efed0ff9640 1 -- 172.21.15.17:0/2686105132 <== mon.2 v1:172.21.15.17:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7efec8083710 con 0x7efed40af900 2024-03-29T11:22:54.754 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.771+0000 7efed0ff9640 10 client.4394.objecter ms_dispatch 0x7efed4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:54.754 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.771+0000 7efed0ff9640 3 client.4394.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:54.754 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.771+0000 7efed0ff9640 3 client.4394.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:54.754 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.771+0000 7efed0ff9640 20 client.4394.objecter dump_active .. 0 homeless 2024-03-29T11:22:54.754 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.773+0000 7efed37fe640 10 monclient: get_auth_request con 0x7efec006e160 auth_method 0 2024-03-29T11:22:54.755 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.775+0000 7efed0ff9640 10 client.4394.objecter ms_handle_connect 0x7efec006e160 2024-03-29T11:22:54.761 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.781+0000 7efed0ff9640 1 -- 172.21.15.17:0/2686105132 <== mon.2 v1:172.21.15.17:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efec804eff0 con 0x7efed40af900 2024-03-29T11:22:54.761 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.781+0000 7efed0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:54.761 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:54.781+0000 7efed0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:55.090 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:55.112+0000 7efeda36b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:55.090 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:55.112+0000 7efeda36b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:55.090 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:55.112+0000 7efeda36b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:22:55.090 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:55.112+0000 7efeda36b640 1 -- 172.21.15.17:0/2686105132 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efeb4005330 con 0x7efed40af900 2024-03-29T11:22:55.091 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:55.113+0000 7efed0ff9640 1 -- 172.21.15.17:0/2686105132 <== mon.2 v1:172.21.15.17:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7efec8050f20 con 0x7efed40af900 2024-03-29T11:22:55.091 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:55.113+0000 7efed0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:55.091 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:55.113+0000 7efed0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:55.092 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:22:55.092 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:55.095 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:55.118+0000 7efeda36b640 10 librados: watch_flush enter 2024-03-29T11:22:55.097 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:55.120+0000 7efeda36b640 10 librados: watch_flush exit 2024-03-29T11:22:55.097 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:55.120+0000 7efeda36b640 20 client.4394.objecter shutdown clearing up homeless session... 2024-03-29T11:22:55.097 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:55.120+0000 7efeda36b640 10 client.4394.objecter successfully canceled tick 2024-03-29T11:22:55.097 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:55.120+0000 7efeda36b640 1 -- 172.21.15.17:0/2686105132 >> v1:172.21.15.17:6816/3236185052 conn(0x7efec006e160 legacy=0x7efec0070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:55.098 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:55.120+0000 7efeda36b640 10 monclient: shutdown 2024-03-29T11:22:55.098 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:55.120+0000 7efeda36b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:55.098 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:55.120+0000 7efeda36b640 1 -- 172.21.15.17:0/2686105132 >> v1:172.21.15.17:6790/0 conn(0x7efed40af900 legacy=0x7efed414feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:55.099 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:55.121+0000 7efeda36b640 1 -- 172.21.15.17:0/2686105132 shutdown_connections 2024-03-29T11:22:55.099 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:55.121+0000 7efeda36b640 1 -- 172.21.15.17:0/2686105132 >> 172.21.15.17:0/2686105132 conn(0x7efed40b35e0 msgr2=0x7efed40b11d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:55.101 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:55.123+0000 7efeda36b640 1 -- 172.21.15.17:0/2686105132 shutdown_connections 2024-03-29T11:22:55.101 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:55.123+0000 7efeda36b640 1 -- 172.21.15.17:0/2686105132 wait complete. 2024-03-29T11:22:55.101 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:22:55.123+0000 7efeda36b640 1 librados: shutdown 2024-03-29T11:22:55.121 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:01.927 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:02.078 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.100+0000 7faae4996640 10 monclient: get_monmap_and_config 2024-03-29T11:23:02.078 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.100+0000 7faae4996640 10 monclient: build_initial_monmap 2024-03-29T11:23:02.078 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.100+0000 7faae4996640 10 monclient: monmap: 2024-03-29T11:23:02.078 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:23:02.078 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:23:02.078 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:23:02.101165+0000 2024-03-29T11:23:02.078 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:23:02.101165+0000 2024-03-29T11:23:02.078 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:02.078 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:02.079 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:23:02.079 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:23:02.079 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:23:02.079 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:02.079 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.101+0000 7faae4996640 1 Processor -- start 2024-03-29T11:23:02.079 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.101+0000 7faae4996640 1 -- start start 2024-03-29T11:23:02.079 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.101+0000 7faae4996640 10 monclient: init 2024-03-29T11:23:02.079 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.102+0000 7faae4996640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:02.079 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.102+0000 7faae4996640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:23:02.079 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.102+0000 7faae4996640 10 monclient(hunting): picked mon.noname-b con 0x7faae00b1f60 addr v1:172.21.15.17:6790/0 2024-03-29T11:23:02.083 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.105+0000 7faae4996640 10 monclient(hunting): picked mon.noname-c con 0x7faae0071eb0 addr v1:172.21.15.94:6789/0 2024-03-29T11:23:02.083 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.105+0000 7faade59c640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7faae00b1f60 0x7faae0143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:58915/0 (socket says 172.21.15.17:58915) 2024-03-29T11:23:02.083 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.105+0000 7faae4996640 10 monclient(hunting): picked mon.noname-a con 0x7faae0143d10 addr v1:172.21.15.17:6789/0 2024-03-29T11:23:02.083 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.105+0000 7faae4996640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faae006ef80 con 0x7faae0143d10 2024-03-29T11:23:02.083 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.105+0000 7faae4996640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faae006f260 con 0x7faae0071eb0 2024-03-29T11:23:02.083 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.105+0000 7faae4996640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faae00b40c0 con 0x7faae00b1f60 2024-03-29T11:23:02.083 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.105+0000 7faae4996640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:02.084 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.105+0000 7faae4996640 10 monclient(hunting): authenticate will time out at 1088.310744s 2024-03-29T11:23:02.084 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.106+0000 7faade59c640 1 -- 172.21.15.17:0/3011798720 learned_addr learned my addr 172.21.15.17:0/3011798720 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:23:02.090 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.109+0000 7faaddd9b640 1 -- 172.21.15.17:0/3011798720 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 778434717 0 0) 0x7faae00b40c0 con 0x7faae00b1f60 2024-03-29T11:23:02.090 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.109+0000 7faaddd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:23:02.090 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.109+0000 7faaddd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:02.090 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.109+0000 7faaddd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:02.090 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.109+0000 7faaddd9b640 10 monclient(hunting): my global_id is 4397 2024-03-29T11:23:02.090 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.109+0000 7faaddd9b640 1 -- 172.21.15.17:0/3011798720 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faacc003520 con 0x7faae00b1f60 2024-03-29T11:23:02.090 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.109+0000 7faaddd9b640 1 -- 172.21.15.17:0/3011798720 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 272879538 0 0) 0x7faae006f260 con 0x7faae0071eb0 2024-03-29T11:23:02.090 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.109+0000 7faaddd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:23:02.090 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.109+0000 7faaddd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:02.090 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.109+0000 7faaddd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:02.090 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.109+0000 7faaddd9b640 10 monclient(hunting): my global_id is 4384 2024-03-29T11:23:02.091 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.109+0000 7faaddd9b640 1 -- 172.21.15.17:0/3011798720 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faae00b40c0 con 0x7faae0071eb0 2024-03-29T11:23:02.091 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.109+0000 7faaddd9b640 1 -- 172.21.15.17:0/3011798720 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1623015406 0 0) 0x7faacc003520 con 0x7faae00b1f60 2024-03-29T11:23:02.091 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.109+0000 7faaddd9b640 1 -- 172.21.15.17:0/3011798720 >> v1:172.21.15.94:6789/0 conn(0x7faae0071eb0 legacy=0x7faae0146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:02.091 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.109+0000 7faaddd9b640 1 -- 172.21.15.17:0/3011798720 >> v1:172.21.15.17:6789/0 conn(0x7faae0143d10 legacy=0x7faae0146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:02.091 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.109+0000 7faaddd9b640 10 monclient: _finish_hunting 0 2024-03-29T11:23:02.091 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.109+0000 7faaddd9b640 1 monclient: found mon.noname-b 2024-03-29T11:23:02.091 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.109+0000 7faaddd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:02.091 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.109+0000 7faaddd9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.17:6790/0 2024-03-29T11:23:02.091 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.109+0000 7faaddd9b640 1 -- 172.21.15.17:0/3011798720 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae0155d70 con 0x7faae00b1f60 2024-03-29T11:23:02.091 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.109+0000 7faaddd9b640 10 monclient: _finish_auth 0 2024-03-29T11:23:02.091 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.109+0000 7faaddd9b640 10 monclient: _check_auth_tickets 2024-03-29T11:23:02.091 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.109+0000 7faaddd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:02.091 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.109+0000 7faaddd9b640 1 -- 172.21.15.17:0/3011798720 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7faad0001690 con 0x7faae00b1f60 2024-03-29T11:23:02.091 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.109+0000 7faaddd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:02.091 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.109+0000 7faaddd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:02.091 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.109+0000 7faaddd9b640 10 monclient: dump: 2024-03-29T11:23:02.092 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:23:02.092 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:23:02.092 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:02.092 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:02.092 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:02.092 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:02.092 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:23:02.092 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:23:02.092 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:23:02.092 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:02.092 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.109+0000 7faae4996640 5 monclient: authenticate success, global_id 4397 2024-03-29T11:23:02.092 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.109+0000 7faae4996640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:02.092 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.110+0000 7faaddd9b640 1 -- 172.21.15.17:0/3011798720 <== mon.2 v1:172.21.15.17:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faad00017f0 con 0x7faae00b1f60 2024-03-29T11:23:02.092 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.110+0000 7faaddd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:02.093 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.110+0000 7faaddd9b640 1 -- 172.21.15.17:0/3011798720 <== mon.2 v1:172.21.15.17:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7faad0001c00 con 0x7faae00b1f60 2024-03-29T11:23:02.093 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.110+0000 7faaddd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:02.093 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.110+0000 7faaddd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:02.093 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.110+0000 7faaddd9b640 10 monclient: dump: 2024-03-29T11:23:02.093 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:23:02.093 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:23:02.093 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:02.093 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:02.093 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:02.093 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:02.093 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:23:02.093 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:23:02.093 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:23:02.093 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:02.093 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.110+0000 7faae4996640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:02.093 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.110+0000 7faae4996640 4 set_mon_vals no callback set 2024-03-29T11:23:02.093 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.111+0000 7faae4996640 10 monclient: shutdown 2024-03-29T11:23:02.094 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.111+0000 7faae4996640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:02.094 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.111+0000 7faae4996640 1 -- 172.21.15.17:0/3011798720 >> v1:172.21.15.17:6790/0 conn(0x7faae00b1f60 legacy=0x7faae0143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:02.094 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.112+0000 7faae4996640 1 -- 172.21.15.17:0/3011798720 shutdown_connections 2024-03-29T11:23:02.094 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.112+0000 7faae4996640 1 -- 172.21.15.17:0/3011798720 >> 172.21.15.17:0/3011798720 conn(0x7faae0066a00 msgr2=0x7faae0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:02.094 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.115+0000 7faae4996640 1 -- 172.21.15.17:0/3011798720 shutdown_connections 2024-03-29T11:23:02.097 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.117+0000 7faae4996640 1 -- 172.21.15.17:0/3011798720 wait complete. 2024-03-29T11:23:02.097 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.118+0000 7faae4996640 10 monclient: build_initial_monmap 2024-03-29T11:23:02.097 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.118+0000 7faae4996640 10 monclient: monmap: 2024-03-29T11:23:02.097 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:23:02.097 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:02.098 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:23:02.098 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:23:02.098 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:02.098 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:02.098 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:23:02.098 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:23:02.099 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:23:02.099 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:02.099 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.118+0000 7faae4996640 1 librados: starting msgr at 2024-03-29T11:23:02.099 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.118+0000 7faae4996640 1 librados: starting objecter 2024-03-29T11:23:02.099 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.119+0000 7faae4996640 1 Processor -- start 2024-03-29T11:23:02.100 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.119+0000 7faae4996640 1 -- start start 2024-03-29T11:23:02.100 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.119+0000 7faae4996640 1 librados: setting wanted keys 2024-03-29T11:23:02.100 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.119+0000 7faae4996640 1 librados: calling monclient init 2024-03-29T11:23:02.100 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.119+0000 7faae4996640 10 monclient: init 2024-03-29T11:23:02.100 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.120+0000 7faae4996640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:02.100 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.120+0000 7faae4996640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:23:02.101 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.120+0000 7faae4996640 10 monclient(hunting): picked mon.noname-c con 0x7faae00b1f60 addr v1:172.21.15.17:6790/0 2024-03-29T11:23:02.101 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.123+0000 7faae4996640 10 monclient(hunting): picked mon.noname-b con 0x7faae0143d10 addr v1:172.21.15.94:6789/0 2024-03-29T11:23:02.102 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.123+0000 7faad5d9b640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7faae0143d10 0x7faae0137cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:49607/0 (socket says 172.21.15.17:49607) 2024-03-29T11:23:02.102 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.124+0000 7faae4996640 10 monclient(hunting): picked mon.noname-a con 0x7faae0154d90 addr v1:172.21.15.17:6789/0 2024-03-29T11:23:02.102 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.124+0000 7faae4996640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faae014b530 con 0x7faae0154d90 2024-03-29T11:23:02.103 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.124+0000 7faae4996640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faae014b7a0 con 0x7faae0143d10 2024-03-29T11:23:02.103 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.124+0000 7faae4996640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faae01405c0 con 0x7faae00b1f60 2024-03-29T11:23:02.103 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.124+0000 7faae4996640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:02.103 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.124+0000 7faae4996640 10 monclient(hunting): authenticate will time out at 1088.329423s 2024-03-29T11:23:02.103 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.124+0000 7faad5d9b640 1 -- 172.21.15.17:0/2458707295 learned_addr learned my addr 172.21.15.17:0/2458707295 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:23:02.104 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.125+0000 7faad77fe640 10 client.?.objecter ms_handle_connect 0x7faae0143d10 2024-03-29T11:23:02.104 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.125+0000 7faad77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:02.104 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.125+0000 7faad77fe640 1 -- 172.21.15.17:0/2458707295 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 617457340 0 0) 0x7faae014b7a0 con 0x7faae0143d10 2024-03-29T11:23:02.104 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.125+0000 7faad77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:23:02.105 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.125+0000 7faad77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:02.105 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.125+0000 7faad77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:02.105 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.125+0000 7faad77fe640 10 monclient(hunting): my global_id is 4387 2024-03-29T11:23:02.105 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.125+0000 7faad77fe640 1 -- 172.21.15.17:0/2458707295 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faac0004eb0 con 0x7faae0143d10 2024-03-29T11:23:02.111 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.127+0000 7faad77fe640 1 -- 172.21.15.17:0/2458707295 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1374956722 0 0) 0x7faac0004eb0 con 0x7faae0143d10 2024-03-29T11:23:02.111 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.127+0000 7faad77fe640 1 -- 172.21.15.17:0/2458707295 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faae014b7a0 con 0x7faae0143d10 2024-03-29T11:23:02.111 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.127+0000 7faad77fe640 10 client.?.objecter ms_handle_connect 0x7faae00b1f60 2024-03-29T11:23:02.112 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.127+0000 7faad77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:02.112 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.127+0000 7faad77fe640 10 client.?.objecter ms_handle_connect 0x7faae0154d90 2024-03-29T11:23:02.112 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.127+0000 7faad77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:02.112 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.127+0000 7faad77fe640 1 -- 172.21.15.17:0/2458707295 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7faac80019b0 con 0x7faae0143d10 2024-03-29T11:23:02.112 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.127+0000 7faad77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:02.112 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.127+0000 7faad77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:02.113 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.127+0000 7faad77fe640 10 monclient(hunting): dump: 2024-03-29T11:23:02.113 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:23:02.113 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:23:02.113 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:02.113 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:02.113 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:02.113 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:02.114 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:23:02.114 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:23:02.114 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:23:02.114 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:02.114 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.127+0000 7faad77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:02.114 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.127+0000 7faad77fe640 1 -- 172.21.15.17:0/2458707295 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3383177400 0 0) 0x7faae014b530 con 0x7faae0154d90 2024-03-29T11:23:02.115 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.127+0000 7faad77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:23:02.115 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.127+0000 7faad77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:02.115 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.127+0000 7faad77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:02.115 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.127+0000 7faad77fe640 10 monclient(hunting): my global_id is 4374 2024-03-29T11:23:02.115 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.127+0000 7faad77fe640 1 -- 172.21.15.17:0/2458707295 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faac0004eb0 con 0x7faae0154d90 2024-03-29T11:23:02.115 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.127+0000 7faad77fe640 1 -- 172.21.15.17:0/2458707295 <== mon.1 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2520126430 0 0) 0x7faae014b7a0 con 0x7faae0143d10 2024-03-29T11:23:02.116 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.127+0000 7faad77fe640 1 -- 172.21.15.17:0/2458707295 >> v1:172.21.15.17:6790/0 conn(0x7faae00b1f60 legacy=0x7faae013fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:02.116 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.127+0000 7faad77fe640 1 -- 172.21.15.17:0/2458707295 >> v1:172.21.15.17:6789/0 conn(0x7faae0154d90 legacy=0x7faae01df000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:02.116 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.127+0000 7faad77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:23:02.116 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.127+0000 7faad77fe640 1 monclient: found mon.b 2024-03-29T11:23:02.116 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.127+0000 7faad77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:02.116 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.127+0000 7faad77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:23:02.117 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.127+0000 7faad77fe640 1 -- 172.21.15.17:0/2458707295 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae01383e0 con 0x7faae0143d10 2024-03-29T11:23:02.117 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.127+0000 7faad77fe640 10 monclient: _finish_auth 0 2024-03-29T11:23:02.117 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.127+0000 7faad77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:23:02.117 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.127+0000 7faad77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:02.117 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.127+0000 7faae4996640 5 monclient: authenticate success, global_id 4387 2024-03-29T11:23:02.117 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.127+0000 7faae4996640 10 monclient: _renew_subs 2024-03-29T11:23:02.118 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.127+0000 7faae4996640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:23:02.118 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.127+0000 7faae4996640 1 -- 172.21.15.17:0/2458707295 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faae0138630 con 0x7faae0143d10 2024-03-29T11:23:02.118 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.127+0000 7faae4996640 10 client.4387.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:02.118 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.127+0000 7faae4996640 10 monclient: _renew_subs 2024-03-29T11:23:02.118 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.127+0000 7faae4996640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:23:02.118 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.127+0000 7faae4996640 1 -- 172.21.15.17:0/2458707295 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faae01df8f0 con 0x7faae0143d10 2024-03-29T11:23:02.119 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.127+0000 7faae4996640 1 librados: init done 2024-03-29T11:23:02.119 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.128+0000 7faad77fe640 1 -- 172.21.15.17:0/2458707295 <== mon.1 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faac8001b10 con 0x7faae0143d10 2024-03-29T11:23:02.119 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.128+0000 7faad77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:02.119 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.128+0000 7faad77fe640 1 -- 172.21.15.17:0/2458707295 <== mon.1 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7faac8003a60 con 0x7faae0143d10 2024-03-29T11:23:02.119 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.128+0000 7faad77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:02.119 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.128+0000 7faad77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:02.120 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.128+0000 7faad77fe640 10 monclient: dump: 2024-03-29T11:23:02.120 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:23:02.120 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:23:02.120 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:02.120 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:02.120 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:02.121 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:02.121 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:23:02.121 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:23:02.121 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:23:02.121 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:02.121 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.128+0000 7faadcd99640 4 set_mon_vals no callback set 2024-03-29T11:23:02.122 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.129+0000 7faad77fe640 1 -- 172.21.15.17:0/2458707295 <== mon.1 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7faac8003bc0 con 0x7faae0143d10 2024-03-29T11:23:02.122 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.130+0000 7faae4996640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:02.122 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.130+0000 7faae4996640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:02.122 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.130+0000 7faae4996640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:23:02.122 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.130+0000 7faae4996640 1 -- 172.21.15.17:0/2458707295 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faac4005080 con 0x7faae0143d10 2024-03-29T11:23:02.123 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.130+0000 7faad77fe640 1 -- 172.21.15.17:0/2458707295 <== mon.1 v1:172.21.15.94:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7faac8084ac0 con 0x7faae0143d10 2024-03-29T11:23:02.123 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.130+0000 7faad77fe640 10 client.4387.objecter ms_dispatch 0x7faae0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:02.123 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.130+0000 7faad77fe640 3 client.4387.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:02.123 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.130+0000 7faad77fe640 3 client.4387.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:02.123 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.130+0000 7faad77fe640 20 client.4387.objecter dump_active .. 0 homeless 2024-03-29T11:23:02.123 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.130+0000 7faade59c640 10 monclient: get_auth_request con 0x7faac006dce0 auth_method 0 2024-03-29T11:23:02.123 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.131+0000 7faad77fe640 10 client.4387.objecter ms_handle_connect 0x7faac006dce0 2024-03-29T11:23:02.124 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.137+0000 7faad77fe640 1 -- 172.21.15.17:0/2458707295 <== mon.1 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faac8050470 con 0x7faae0143d10 2024-03-29T11:23:02.124 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.137+0000 7faad77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:02.124 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.137+0000 7faad77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:02.452 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.474+0000 7faae4996640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:02.453 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.474+0000 7faae4996640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:02.453 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.474+0000 7faae4996640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:23:02.453 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.474+0000 7faae4996640 1 -- 172.21.15.17:0/2458707295 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faac4005330 con 0x7faae0143d10 2024-03-29T11:23:02.456 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.478+0000 7faad77fe640 1 -- 172.21.15.17:0/2458707295 <== mon.1 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7faac8084170 con 0x7faae0143d10 2024-03-29T11:23:02.456 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.478+0000 7faad77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:02.456 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.478+0000 7faad77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:02.457 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:23:02.457 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:02.460 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.482+0000 7faae4996640 10 librados: watch_flush enter 2024-03-29T11:23:02.460 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.482+0000 7faae4996640 10 librados: watch_flush exit 2024-03-29T11:23:02.460 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.482+0000 7faae4996640 20 client.4387.objecter shutdown clearing up homeless session... 2024-03-29T11:23:02.460 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.482+0000 7faae4996640 10 client.4387.objecter successfully canceled tick 2024-03-29T11:23:02.460 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.482+0000 7faae4996640 1 -- 172.21.15.17:0/2458707295 >> v1:172.21.15.17:6816/3236185052 conn(0x7faac006dce0 legacy=0x7faac00701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:02.460 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.482+0000 7faae4996640 10 monclient: shutdown 2024-03-29T11:23:02.460 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.482+0000 7faae4996640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:02.461 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.482+0000 7faae4996640 1 -- 172.21.15.17:0/2458707295 >> v1:172.21.15.94:6789/0 conn(0x7faae0143d10 legacy=0x7faae0137cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:02.461 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.482+0000 7faae4996640 1 -- 172.21.15.17:0/2458707295 shutdown_connections 2024-03-29T11:23:02.461 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.482+0000 7faae4996640 1 -- 172.21.15.17:0/2458707295 >> 172.21.15.17:0/2458707295 conn(0x7faae0066a00 msgr2=0x7faae00b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:02.462 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.485+0000 7faae4996640 1 -- 172.21.15.17:0/2458707295 shutdown_connections 2024-03-29T11:23:02.463 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.485+0000 7faae4996640 1 -- 172.21.15.17:0/2458707295 wait complete. 2024-03-29T11:23:02.463 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:02.485+0000 7faae4996640 1 librados: shutdown 2024-03-29T11:23:02.482 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:09.289 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:09.444 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.466+0000 7f4bc460d640 10 monclient: get_monmap_and_config 2024-03-29T11:23:09.444 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.466+0000 7f4bc460d640 10 monclient: build_initial_monmap 2024-03-29T11:23:09.444 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.466+0000 7f4bc460d640 10 monclient: monmap: 2024-03-29T11:23:09.445 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:23:09.445 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:23:09.445 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:23:09.467327+0000 2024-03-29T11:23:09.445 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:23:09.467327+0000 2024-03-29T11:23:09.445 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:09.445 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:09.445 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:23:09.445 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:23:09.445 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:23:09.445 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:09.446 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.467+0000 7f4bc460d640 1 Processor -- start 2024-03-29T11:23:09.447 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.468+0000 7f4bc460d640 1 -- start start 2024-03-29T11:23:09.447 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.468+0000 7f4bc460d640 10 monclient: init 2024-03-29T11:23:09.448 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.469+0000 7f4bc460d640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:09.448 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.469+0000 7f4bc460d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:23:09.448 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.469+0000 7f4bc460d640 10 monclient(hunting): picked mon.noname-c con 0x7f4bbc0b1f60 addr v1:172.21.15.94:6789/0 2024-03-29T11:23:09.448 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.469+0000 7f4bc460d640 10 monclient(hunting): picked mon.noname-a con 0x7f4bbc071eb0 addr v1:172.21.15.17:6789/0 2024-03-29T11:23:09.448 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.469+0000 7f4bc460d640 10 monclient(hunting): picked mon.noname-b con 0x7f4bbc143d10 addr v1:172.21.15.17:6790/0 2024-03-29T11:23:09.448 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.469+0000 7f4bc460d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bbc06ef80 con 0x7f4bbc071eb0 2024-03-29T11:23:09.448 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.469+0000 7f4bc460d640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bbc06f260 con 0x7f4bbc0b1f60 2024-03-29T11:23:09.448 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.469+0000 7f4bc460d640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bbc0b40c0 con 0x7f4bbc143d10 2024-03-29T11:23:09.448 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.469+0000 7f4bc460d640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:09.448 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.469+0000 7f4bc460d640 10 monclient(hunting): authenticate will time out at 1095.674387s 2024-03-29T11:23:09.452 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.472+0000 7f4bc1ba8640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f4bbc071eb0 0x7f4bbc146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:59797/0 (socket says 172.21.15.17:59797) 2024-03-29T11:23:09.452 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.472+0000 7f4bc1ba8640 1 -- 172.21.15.17:0/2033265468 learned_addr learned my addr 172.21.15.17:0/2033265468 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:23:09.457 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.477+0000 7f4bc13a7640 1 -- 172.21.15.17:0/2033265468 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1369309016 0 0) 0x7f4bbc06f260 con 0x7f4bbc0b1f60 2024-03-29T11:23:09.457 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.477+0000 7f4bc13a7640 10 monclient(hunting): _negotiate 2024-03-29T11:23:09.457 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.477+0000 7f4bc13a7640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:09.457 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.477+0000 7f4bc13a7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:09.457 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.477+0000 7f4bc13a7640 10 monclient(hunting): my global_id is 4390 2024-03-29T11:23:09.457 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.477+0000 7f4bc13a7640 1 -- 172.21.15.17:0/2033265468 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ba8003520 con 0x7f4bbc0b1f60 2024-03-29T11:23:09.457 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.477+0000 7f4bc13a7640 1 -- 172.21.15.17:0/2033265468 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 952567281 0 0) 0x7f4bbc06ef80 con 0x7f4bbc071eb0 2024-03-29T11:23:09.457 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.477+0000 7f4bc13a7640 10 monclient(hunting): _negotiate 2024-03-29T11:23:09.457 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.477+0000 7f4bc13a7640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:09.458 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.477+0000 7f4bc13a7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:09.458 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.477+0000 7f4bc13a7640 10 monclient(hunting): my global_id is 4377 2024-03-29T11:23:09.458 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.477+0000 7f4bc13a7640 1 -- 172.21.15.17:0/2033265468 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bbc06f260 con 0x7f4bbc071eb0 2024-03-29T11:23:09.459 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.479+0000 7f4bc13a7640 1 -- 172.21.15.17:0/2033265468 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2419236368 0 0) 0x7f4bbc06f260 con 0x7f4bbc071eb0 2024-03-29T11:23:09.459 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.479+0000 7f4bc13a7640 1 -- 172.21.15.17:0/2033265468 >> v1:172.21.15.17:6790/0 conn(0x7f4bbc143d10 legacy=0x7f4bbc146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:09.459 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.479+0000 7f4bc13a7640 1 -- 172.21.15.17:0/2033265468 >> v1:172.21.15.94:6789/0 conn(0x7f4bbc0b1f60 legacy=0x7f4bbc143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:09.459 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.479+0000 7f4bc13a7640 10 monclient: _finish_hunting 0 2024-03-29T11:23:09.459 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.479+0000 7f4bc13a7640 1 monclient: found mon.noname-a 2024-03-29T11:23:09.459 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.479+0000 7f4bc13a7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:09.459 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.479+0000 7f4bc13a7640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.17:6789/0 2024-03-29T11:23:09.459 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.479+0000 7f4bc13a7640 1 -- 172.21.15.17:0/2033265468 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bbc155d70 con 0x7f4bbc071eb0 2024-03-29T11:23:09.459 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.479+0000 7f4bc13a7640 10 monclient: _finish_auth 0 2024-03-29T11:23:09.459 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.479+0000 7f4bc13a7640 10 monclient: _check_auth_tickets 2024-03-29T11:23:09.460 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.479+0000 7f4bc13a7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:09.460 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.479+0000 7f4bc13a7640 1 -- 172.21.15.17:0/2033265468 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f4bb40018a0 con 0x7f4bbc071eb0 2024-03-29T11:23:09.460 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.479+0000 7f4bc13a7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:09.460 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.479+0000 7f4bc13a7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:09.460 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.479+0000 7f4bc13a7640 10 monclient: dump: 2024-03-29T11:23:09.460 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:23:09.460 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:23:09.460 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:09.460 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:09.460 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:09.460 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:09.460 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:23:09.461 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:23:09.461 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:23:09.461 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:09.461 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.479+0000 7f4bc460d640 5 monclient: authenticate success, global_id 4377 2024-03-29T11:23:09.461 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.479+0000 7f4bc460d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:09.461 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.479+0000 7f4bc13a7640 1 -- 172.21.15.17:0/2033265468 <== mon.0 v1:172.21.15.17:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4bb4001a00 con 0x7f4bbc071eb0 2024-03-29T11:23:09.461 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.479+0000 7f4bc13a7640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:09.461 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.479+0000 7f4bc13a7640 1 -- 172.21.15.17:0/2033265468 <== mon.0 v1:172.21.15.17:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f4bb4001e10 con 0x7f4bbc071eb0 2024-03-29T11:23:09.461 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.479+0000 7f4bc13a7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:09.461 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.479+0000 7f4bc13a7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:09.461 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.479+0000 7f4bc13a7640 10 monclient: dump: 2024-03-29T11:23:09.461 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:23:09.461 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:23:09.462 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:09.462 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:09.462 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:09.462 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:09.462 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:23:09.462 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:23:09.462 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:23:09.462 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:09.462 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.479+0000 7f4bc460d640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:09.462 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.479+0000 7f4bc460d640 4 set_mon_vals no callback set 2024-03-29T11:23:09.462 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.480+0000 7f4bc460d640 10 monclient: shutdown 2024-03-29T11:23:09.462 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.480+0000 7f4bc460d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:09.463 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.480+0000 7f4bc460d640 1 -- 172.21.15.17:0/2033265468 >> v1:172.21.15.17:6789/0 conn(0x7f4bbc071eb0 legacy=0x7f4bbc146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:09.463 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.481+0000 7f4bc460d640 1 -- 172.21.15.17:0/2033265468 shutdown_connections 2024-03-29T11:23:09.463 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.481+0000 7f4bc460d640 1 -- 172.21.15.17:0/2033265468 >> 172.21.15.17:0/2033265468 conn(0x7f4bbc066a00 msgr2=0x7f4bbc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:09.470 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.481+0000 7f4bc460d640 1 -- 172.21.15.17:0/2033265468 shutdown_connections 2024-03-29T11:23:09.470 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.481+0000 7f4bc460d640 1 -- 172.21.15.17:0/2033265468 wait complete. 2024-03-29T11:23:09.470 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.482+0000 7f4bc460d640 10 monclient: build_initial_monmap 2024-03-29T11:23:09.470 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.482+0000 7f4bc460d640 10 monclient: monmap: 2024-03-29T11:23:09.470 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:23:09.470 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:09.470 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:23:09.470 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:23:09.470 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:09.471 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:09.471 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:23:09.471 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:23:09.471 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:23:09.471 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:09.471 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.482+0000 7f4bc460d640 1 librados: starting msgr at 2024-03-29T11:23:09.471 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.482+0000 7f4bc460d640 1 librados: starting objecter 2024-03-29T11:23:09.471 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.483+0000 7f4bc460d640 1 Processor -- start 2024-03-29T11:23:09.493 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.483+0000 7f4bc460d640 1 -- start start 2024-03-29T11:23:09.494 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.483+0000 7f4bc460d640 1 librados: setting wanted keys 2024-03-29T11:23:09.494 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.483+0000 7f4bc460d640 1 librados: calling monclient init 2024-03-29T11:23:09.494 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.483+0000 7f4bc460d640 10 monclient: init 2024-03-29T11:23:09.494 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.483+0000 7f4bc460d640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:09.494 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.483+0000 7f4bc460d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:23:09.494 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.484+0000 7f4bc460d640 10 monclient(hunting): picked mon.noname-a con 0x7f4bbc0b1f60 addr v1:172.21.15.17:6789/0 2024-03-29T11:23:09.494 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.484+0000 7f4bc460d640 10 monclient(hunting): picked mon.noname-c con 0x7f4bbc143d10 addr v1:172.21.15.17:6790/0 2024-03-29T11:23:09.494 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.484+0000 7f4bc1ba8640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f4bbc143d10 0x7f4bbc131920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:51076/0 (socket says 172.21.15.17:51076) 2024-03-29T11:23:09.494 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.484+0000 7f4bc1ba8640 1 -- 172.21.15.17:0/3794469970 learned_addr learned my addr 172.21.15.17:0/3794469970 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:23:09.494 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.484+0000 7f4bb2ffd640 10 client.?.objecter ms_handle_connect 0x7f4bbc143d10 2024-03-29T11:23:09.494 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.484+0000 7f4bb2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:09.495 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.484+0000 7f4bc460d640 10 monclient(hunting): picked mon.noname-b con 0x7f4bbc133a90 addr v1:172.21.15.94:6789/0 2024-03-29T11:23:09.495 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.484+0000 7f4bc460d640 1 -- 172.21.15.17:0/3794469970 --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bbc132740 con 0x7f4bbc0b1f60 2024-03-29T11:23:09.495 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.484+0000 7f4bc460d640 1 -- 172.21.15.17:0/3794469970 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bbc071830 con 0x7f4bbc133a90 2024-03-29T11:23:09.495 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.484+0000 7f4bc460d640 1 -- 172.21.15.17:0/3794469970 --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bbc070e30 con 0x7f4bbc143d10 2024-03-29T11:23:09.495 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.484+0000 7f4bc460d640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:09.495 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.484+0000 7f4bc460d640 10 monclient(hunting): authenticate will time out at 1095.689854s 2024-03-29T11:23:09.495 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.485+0000 7f4bb2ffd640 10 client.?.objecter ms_handle_connect 0x7f4bbc0b1f60 2024-03-29T11:23:09.495 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.485+0000 7f4bb2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:09.495 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.485+0000 7f4bb2ffd640 1 -- 172.21.15.17:0/3794469970 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2912032394 0 0) 0x7f4bbc070e30 con 0x7f4bbc143d10 2024-03-29T11:23:09.495 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.485+0000 7f4bb2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:09.495 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.485+0000 7f4bb2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:09.495 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.485+0000 7f4bb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:09.495 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.485+0000 7f4bb2ffd640 10 monclient(hunting): my global_id is 4403 2024-03-29T11:23:09.495 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.485+0000 7f4bb2ffd640 1 -- 172.21.15.17:0/3794469970 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ba0004eb0 con 0x7f4bbc143d10 2024-03-29T11:23:09.496 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.485+0000 7f4bb2ffd640 10 client.?.objecter ms_handle_connect 0x7f4bbc133a90 2024-03-29T11:23:09.496 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.485+0000 7f4bb2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:09.496 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.485+0000 7f4bb2ffd640 1 -- 172.21.15.17:0/3794469970 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 978812255 0 0) 0x7f4bbc132740 con 0x7f4bbc0b1f60 2024-03-29T11:23:09.496 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.485+0000 7f4bb2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:09.496 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.485+0000 7f4bb2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:09.496 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.485+0000 7f4bb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:09.496 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.485+0000 7f4bb2ffd640 10 monclient(hunting): my global_id is 4380 2024-03-29T11:23:09.496 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.485+0000 7f4bb2ffd640 1 -- 172.21.15.17:0/3794469970 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bbc070e30 con 0x7f4bbc0b1f60 2024-03-29T11:23:09.496 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.485+0000 7f4bb2ffd640 1 -- 172.21.15.17:0/3794469970 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2369258638 0 0) 0x7f4bbc071830 con 0x7f4bbc133a90 2024-03-29T11:23:09.496 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.485+0000 7f4bb2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:09.496 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.485+0000 7f4bb2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:09.496 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.485+0000 7f4bb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:09.496 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.485+0000 7f4bb2ffd640 10 monclient(hunting): my global_id is 4393 2024-03-29T11:23:09.496 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.485+0000 7f4bb2ffd640 1 -- 172.21.15.17:0/3794469970 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bbc132740 con 0x7f4bbc133a90 2024-03-29T11:23:09.497 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.485+0000 7f4bb2ffd640 1 -- 172.21.15.17:0/3794469970 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3622375914 0 0) 0x7f4bbc070e30 con 0x7f4bbc0b1f60 2024-03-29T11:23:09.497 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.485+0000 7f4bb2ffd640 1 -- 172.21.15.17:0/3794469970 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ba0005620 con 0x7f4bbc0b1f60 2024-03-29T11:23:09.497 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.485+0000 7f4bb2ffd640 1 -- 172.21.15.17:0/3794469970 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1018169896 0 0) 0x7f4ba0004eb0 con 0x7f4bbc143d10 2024-03-29T11:23:09.497 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.485+0000 7f4bb2ffd640 1 -- 172.21.15.17:0/3794469970 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ba0005be0 con 0x7f4bbc143d10 2024-03-29T11:23:09.497 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.485+0000 7f4bb2ffd640 1 -- 172.21.15.17:0/3794469970 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f4bb4001dd0 con 0x7f4bbc143d10 2024-03-29T11:23:09.497 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.485+0000 7f4bb2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:09.497 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.485+0000 7f4bb2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:09.497 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.485+0000 7f4bb2ffd640 10 monclient(hunting): dump: 2024-03-29T11:23:09.497 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:23:09.497 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:23:09.497 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:09.497 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:09.497 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:09.498 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:09.498 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:23:09.498 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:23:09.498 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:23:09.498 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:09.498 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.485+0000 7f4bb2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:09.498 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.486+0000 7f4bb2ffd640 1 -- 172.21.15.17:0/3794469970 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f4bb80018d0 con 0x7f4bbc0b1f60 2024-03-29T11:23:09.498 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.486+0000 7f4bb2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:09.498 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.486+0000 7f4bb2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:09.498 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.486+0000 7f4bb2ffd640 10 monclient(hunting): dump: 2024-03-29T11:23:09.498 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:23:09.498 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:23:09.498 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:09.498 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:09.499 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:09.499 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:09.499 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:23:09.499 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:23:09.499 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:23:09.499 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:09.499 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.486+0000 7f4bb2ffd640 1 -- 172.21.15.17:0/3794469970 <== mon.2 v1:172.21.15.17:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3267141692 0 0) 0x7f4ba0005be0 con 0x7f4bbc143d10 2024-03-29T11:23:09.499 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.486+0000 7f4bb2ffd640 1 -- 172.21.15.17:0/3794469970 >> v1:172.21.15.94:6789/0 conn(0x7f4bbc133a90 legacy=0x7f4bbc132030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:09.499 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.486+0000 7f4bb2ffd640 1 -- 172.21.15.17:0/3794469970 >> v1:172.21.15.17:6789/0 conn(0x7f4bbc0b1f60 legacy=0x7f4bbc1534e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:09.499 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.486+0000 7f4bb2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:23:09.499 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.486+0000 7f4bb2ffd640 1 monclient: found mon.c 2024-03-29T11:23:09.499 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.486+0000 7f4bb2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:09.499 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.486+0000 7f4bb2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:23:09.499 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.486+0000 7f4bb2ffd640 1 -- 172.21.15.17:0/3794469970 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bbc071110 con 0x7f4bbc143d10 2024-03-29T11:23:09.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.486+0000 7f4bb2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:23:09.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.486+0000 7f4bb2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:23:09.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.486+0000 7f4bb2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:09.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.486+0000 7f4bc460d640 5 monclient: authenticate success, global_id 4403 2024-03-29T11:23:09.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.486+0000 7f4bc460d640 10 monclient: _renew_subs 2024-03-29T11:23:09.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.486+0000 7f4bc460d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:23:09.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.486+0000 7f4bc460d640 1 -- 172.21.15.17:0/3794469970 --> v1:172.21.15.17:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bbc06f720 con 0x7f4bbc143d10 2024-03-29T11:23:09.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.486+0000 7f4bc460d640 10 client.4403.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:09.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.486+0000 7f4bc460d640 10 monclient: _renew_subs 2024-03-29T11:23:09.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.486+0000 7f4bc460d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:23:09.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.486+0000 7f4bc460d640 1 -- 172.21.15.17:0/3794469970 --> v1:172.21.15.17:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bbc070120 con 0x7f4bbc143d10 2024-03-29T11:23:09.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.486+0000 7f4bc460d640 1 librados: init done 2024-03-29T11:23:09.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.487+0000 7f4bb2ffd640 1 -- 172.21.15.17:0/3794469970 <== mon.2 v1:172.21.15.17:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4bb4004e10 con 0x7f4bbc143d10 2024-03-29T11:23:09.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.487+0000 7f4bb2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:09.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.487+0000 7f4bb2ffd640 1 -- 172.21.15.17:0/3794469970 <== mon.2 v1:172.21.15.17:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f4bb4005120 con 0x7f4bbc143d10 2024-03-29T11:23:09.501 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.487+0000 7f4bb2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:09.501 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.487+0000 7f4bb2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:09.501 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.487+0000 7f4bb2ffd640 10 monclient: dump: 2024-03-29T11:23:09.501 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:23:09.501 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:23:09.501 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:09.501 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:09.501 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:09.501 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:09.501 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:23:09.501 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:23:09.502 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:23:09.502 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:09.502 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.487+0000 7f4bb3fff640 4 set_mon_vals no callback set 2024-03-29T11:23:09.502 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.488+0000 7f4bc460d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:09.502 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.488+0000 7f4bc460d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:09.502 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.488+0000 7f4bc460d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:23:09.502 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.488+0000 7f4bc460d640 1 -- 172.21.15.17:0/3794469970 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ba4005080 con 0x7f4bbc143d10 2024-03-29T11:23:09.502 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.489+0000 7f4bb2ffd640 1 -- 172.21.15.17:0/3794469970 <== mon.2 v1:172.21.15.17:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f4bb401b0f0 con 0x7f4bbc143d10 2024-03-29T11:23:09.502 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.490+0000 7f4bb2ffd640 1 -- 172.21.15.17:0/3794469970 <== mon.2 v1:172.21.15.17:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f4bb4084ec0 con 0x7f4bbc143d10 2024-03-29T11:23:09.502 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.490+0000 7f4bb2ffd640 10 client.4403.objecter ms_dispatch 0x7f4bbc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:09.502 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.490+0000 7f4bb2ffd640 3 client.4403.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:09.502 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.490+0000 7f4bb2ffd640 3 client.4403.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:09.502 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.490+0000 7f4bb2ffd640 20 client.4403.objecter dump_active .. 0 homeless 2024-03-29T11:23:09.502 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.491+0000 7f4bc23a9640 10 monclient: get_auth_request con 0x7f4ba006e300 auth_method 0 2024-03-29T11:23:09.503 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.491+0000 7f4bb2ffd640 10 client.4403.objecter ms_handle_connect 0x7f4ba006e300 2024-03-29T11:23:09.503 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.496+0000 7f4bb2ffd640 1 -- 172.21.15.17:0/3794469970 <== mon.2 v1:172.21.15.17:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4bb40505d0 con 0x7f4bbc143d10 2024-03-29T11:23:09.503 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.496+0000 7f4bb2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:09.503 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.496+0000 7f4bb2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:09.863 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.885+0000 7f4bc460d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:09.863 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.885+0000 7f4bc460d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:09.863 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.885+0000 7f4bc460d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:23:09.863 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.885+0000 7f4bc460d640 1 -- 172.21.15.17:0/3794469970 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ba4005330 con 0x7f4bbc143d10 2024-03-29T11:23:09.864 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.886+0000 7f4bb2ffd640 1 -- 172.21.15.17:0/3794469970 <== mon.2 v1:172.21.15.17:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f4bb40526d0 con 0x7f4bbc143d10 2024-03-29T11:23:09.864 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.886+0000 7f4bb2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:09.864 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.886+0000 7f4bb2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:09.865 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:23:09.865 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:09.868 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.889+0000 7f4bc460d640 10 librados: watch_flush enter 2024-03-29T11:23:09.868 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.889+0000 7f4bc460d640 10 librados: watch_flush exit 2024-03-29T11:23:09.869 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.889+0000 7f4bc460d640 20 client.4403.objecter shutdown clearing up homeless session... 2024-03-29T11:23:09.869 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.889+0000 7f4bc460d640 10 client.4403.objecter successfully canceled tick 2024-03-29T11:23:09.869 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.889+0000 7f4bc460d640 1 -- 172.21.15.17:0/3794469970 >> v1:172.21.15.17:6816/3236185052 conn(0x7f4ba006e300 legacy=0x7f4ba00707c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:09.869 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.889+0000 7f4bc460d640 10 monclient: shutdown 2024-03-29T11:23:09.869 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.889+0000 7f4bc460d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:09.869 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.889+0000 7f4bc460d640 1 -- 172.21.15.17:0/3794469970 >> v1:172.21.15.17:6790/0 conn(0x7f4bbc143d10 legacy=0x7f4bbc131920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:09.869 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.890+0000 7f4bc460d640 1 -- 172.21.15.17:0/3794469970 shutdown_connections 2024-03-29T11:23:09.869 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.890+0000 7f4bc460d640 1 -- 172.21.15.17:0/3794469970 >> 172.21.15.17:0/3794469970 conn(0x7f4bbc066a00 msgr2=0x7f4bbc146980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:09.869 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.890+0000 7f4bc460d640 1 -- 172.21.15.17:0/3794469970 shutdown_connections 2024-03-29T11:23:09.869 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.890+0000 7f4bc460d640 1 -- 172.21.15.17:0/3794469970 wait complete. 2024-03-29T11:23:09.869 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:09.890+0000 7f4bc460d640 1 librados: shutdown 2024-03-29T11:23:09.882 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:16.689 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:16.853 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.873+0000 7fe5a93d6640 10 monclient: get_monmap_and_config 2024-03-29T11:23:16.853 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.873+0000 7fe5a93d6640 10 monclient: build_initial_monmap 2024-03-29T11:23:16.853 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.873+0000 7fe5a93d6640 10 monclient: monmap: 2024-03-29T11:23:16.853 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:23:16.853 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:23:16.853 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:23:16.874278+0000 2024-03-29T11:23:16.853 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:23:16.874278+0000 2024-03-29T11:23:16.853 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:16.853 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:16.853 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:23:16.853 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:23:16.853 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:23:16.854 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:16.854 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.874+0000 7fe5a93d6640 1 Processor -- start 2024-03-29T11:23:16.854 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.874+0000 7fe5a93d6640 1 -- start start 2024-03-29T11:23:16.854 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.874+0000 7fe5a93d6640 10 monclient: init 2024-03-29T11:23:16.855 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.875+0000 7fe5a93d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:16.855 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.875+0000 7fe5a93d6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:23:16.855 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.875+0000 7fe5a93d6640 10 monclient(hunting): picked mon.noname-b con 0x7fe5a40ae250 addr v1:172.21.15.17:6790/0 2024-03-29T11:23:16.855 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.875+0000 7fe5a93d6640 10 monclient(hunting): picked mon.noname-c con 0x7fe5a413f970 addr v1:172.21.15.94:6789/0 2024-03-29T11:23:16.855 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.875+0000 7fe5a93d6640 10 monclient(hunting): picked mon.noname-a con 0x7fe5a4142800 addr v1:172.21.15.17:6789/0 2024-03-29T11:23:16.855 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.875+0000 7fe5a93d6640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a40afe00 con 0x7fe5a4142800 2024-03-29T11:23:16.855 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.875+0000 7fe5a93d6640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a4142c10 con 0x7fe5a413f970 2024-03-29T11:23:16.855 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.875+0000 7fe5a93d6640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a40afb20 con 0x7fe5a40ae250 2024-03-29T11:23:16.855 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.875+0000 7fe5a93d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:16.855 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.875+0000 7fe5a93d6640 10 monclient(hunting): authenticate will time out at 1103.080420s 2024-03-29T11:23:16.857 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.878+0000 7fe5a27fc640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fe5a413f970 0x7fe5a41420f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:49511/0 (socket says 172.21.15.17:49511) 2024-03-29T11:23:16.857 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.878+0000 7fe5a27fc640 1 -- 172.21.15.17:0/1185518250 learned_addr learned my addr 172.21.15.17:0/1185518250 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:23:16.860 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.882+0000 7fe5a1ffb640 1 -- 172.21.15.17:0/1185518250 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3554890523 0 0) 0x7fe5a40afe00 con 0x7fe5a4142800 2024-03-29T11:23:16.864 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.882+0000 7fe5a1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:16.864 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.882+0000 7fe5a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:16.865 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.882+0000 7fe5a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:16.865 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.882+0000 7fe5a1ffb640 10 monclient(hunting): my global_id is 4383 2024-03-29T11:23:16.865 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.882+0000 7fe5a1ffb640 1 -- 172.21.15.17:0/1185518250 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe58c003520 con 0x7fe5a4142800 2024-03-29T11:23:16.865 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.882+0000 7fe5a1ffb640 1 -- 172.21.15.17:0/1185518250 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4246119817 0 0) 0x7fe5a4142c10 con 0x7fe5a413f970 2024-03-29T11:23:16.865 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.882+0000 7fe5a1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:16.865 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.882+0000 7fe5a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:16.865 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.882+0000 7fe5a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:16.865 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.882+0000 7fe5a1ffb640 10 monclient(hunting): my global_id is 4396 2024-03-29T11:23:16.865 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.882+0000 7fe5a1ffb640 1 -- 172.21.15.17:0/1185518250 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5a40afe00 con 0x7fe5a413f970 2024-03-29T11:23:16.865 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.883+0000 7fe5a1ffb640 1 -- 172.21.15.17:0/1185518250 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4262870882 0 0) 0x7fe58c003520 con 0x7fe5a4142800 2024-03-29T11:23:16.865 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.883+0000 7fe5a1ffb640 1 -- 172.21.15.17:0/1185518250 >> v1:172.21.15.17:6790/0 conn(0x7fe5a40ae250 legacy=0x7fe5a413f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:16.865 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.883+0000 7fe5a1ffb640 1 -- 172.21.15.17:0/1185518250 >> v1:172.21.15.94:6789/0 conn(0x7fe5a413f970 legacy=0x7fe5a41420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:16.865 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.883+0000 7fe5a1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:23:16.866 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.883+0000 7fe5a1ffb640 1 monclient: found mon.noname-a 2024-03-29T11:23:16.866 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.883+0000 7fe5a1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:16.866 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.883+0000 7fe5a1ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.17:6789/0 2024-03-29T11:23:16.866 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.883+0000 7fe5a1ffb640 1 -- 172.21.15.17:0/1185518250 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a40aab80 con 0x7fe5a4142800 2024-03-29T11:23:16.866 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.883+0000 7fe5a1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:23:16.866 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.883+0000 7fe5a1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:23:16.866 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.883+0000 7fe5a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:16.866 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.883+0000 7fe5a1ffb640 1 -- 172.21.15.17:0/1185518250 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fe598001690 con 0x7fe5a4142800 2024-03-29T11:23:16.866 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.883+0000 7fe5a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:16.866 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.883+0000 7fe5a1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:16.866 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.883+0000 7fe5a1ffb640 10 monclient: dump: 2024-03-29T11:23:16.866 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:23:16.866 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:23:16.866 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:16.866 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:16.866 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:16.867 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:16.867 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:23:16.867 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:23:16.867 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:23:16.867 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:16.867 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.884+0000 7fe5a93d6640 5 monclient: authenticate success, global_id 4383 2024-03-29T11:23:16.867 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.884+0000 7fe5a93d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:16.867 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.887+0000 7fe5a1ffb640 1 -- 172.21.15.17:0/1185518250 <== mon.0 v1:172.21.15.17:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe598001870 con 0x7fe5a4142800 2024-03-29T11:23:16.867 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.887+0000 7fe5a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:16.867 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.887+0000 7fe5a1ffb640 1 -- 172.21.15.17:0/1185518250 <== mon.0 v1:172.21.15.17:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fe598001690 con 0x7fe5a4142800 2024-03-29T11:23:16.867 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.887+0000 7fe5a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:16.868 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.887+0000 7fe5a1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:16.868 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.887+0000 7fe5a1ffb640 10 monclient: dump: 2024-03-29T11:23:16.868 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:23:16.868 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:23:16.868 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:16.868 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:16.868 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:16.868 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:16.868 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:23:16.868 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:23:16.868 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:23:16.868 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:16.868 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.887+0000 7fe5a93d6640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:16.868 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.887+0000 7fe5a93d6640 4 set_mon_vals no callback set 2024-03-29T11:23:16.868 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.888+0000 7fe5a93d6640 10 monclient: shutdown 2024-03-29T11:23:16.868 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.888+0000 7fe5a93d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:16.869 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.888+0000 7fe5a93d6640 1 -- 172.21.15.17:0/1185518250 >> v1:172.21.15.17:6789/0 conn(0x7fe5a4142800 legacy=0x7fe5a414da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:16.869 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.889+0000 7fe5a93d6640 1 -- 172.21.15.17:0/1185518250 shutdown_connections 2024-03-29T11:23:16.870 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.889+0000 7fe5a93d6640 1 -- 172.21.15.17:0/1185518250 >> 172.21.15.17:0/1185518250 conn(0x7fe5a40b35e0 msgr2=0x7fe5a40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:16.870 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.889+0000 7fe5a93d6640 1 -- 172.21.15.17:0/1185518250 shutdown_connections 2024-03-29T11:23:16.870 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.889+0000 7fe5a93d6640 1 -- 172.21.15.17:0/1185518250 wait complete. 2024-03-29T11:23:16.870 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.890+0000 7fe5a93d6640 10 monclient: build_initial_monmap 2024-03-29T11:23:16.870 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.890+0000 7fe5a93d6640 10 monclient: monmap: 2024-03-29T11:23:16.870 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:23:16.870 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:16.870 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:23:16.870 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:23:16.870 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:16.870 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:16.870 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:23:16.870 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:23:16.871 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:23:16.871 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:16.871 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.891+0000 7fe5a93d6640 1 librados: starting msgr at 2024-03-29T11:23:16.871 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.891+0000 7fe5a93d6640 1 librados: starting objecter 2024-03-29T11:23:16.871 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.891+0000 7fe5a93d6640 1 Processor -- start 2024-03-29T11:23:16.872 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.891+0000 7fe5a93d6640 1 -- start start 2024-03-29T11:23:16.872 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.891+0000 7fe5a93d6640 1 librados: setting wanted keys 2024-03-29T11:23:16.872 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.891+0000 7fe5a93d6640 1 librados: calling monclient init 2024-03-29T11:23:16.872 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.891+0000 7fe5a93d6640 10 monclient: init 2024-03-29T11:23:16.873 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.892+0000 7fe5a93d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:16.873 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.892+0000 7fe5a93d6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:23:16.873 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.892+0000 7fe5a93d6640 10 monclient(hunting): picked mon.noname-c con 0x7fe5a40ae250 addr v1:172.21.15.17:6790/0 2024-03-29T11:23:16.873 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.892+0000 7fe5a93d6640 10 monclient(hunting): picked mon.noname-a con 0x7fe5a413f970 addr v1:172.21.15.17:6789/0 2024-03-29T11:23:16.873 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.892+0000 7fe5a93d6640 10 monclient(hunting): picked mon.noname-b con 0x7fe5a4142800 addr v1:172.21.15.94:6789/0 2024-03-29T11:23:16.873 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.892+0000 7fe5a93d6640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a4145250 con 0x7fe5a413f970 2024-03-29T11:23:16.873 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.892+0000 7fe5a93d6640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a4146420 con 0x7fe5a4142800 2024-03-29T11:23:16.873 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.892+0000 7fe5a93d6640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a406e7a0 con 0x7fe5a40ae250 2024-03-29T11:23:16.873 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.892+0000 7fe5a93d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:16.873 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.892+0000 7fe5a93d6640 10 monclient(hunting): authenticate will time out at 1103.097651s 2024-03-29T11:23:16.878 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.899+0000 7fe5a27fc640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fe5a413f970 0x7fe5a413abc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:37213/0 (socket says 172.21.15.17:37213) 2024-03-29T11:23:16.878 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.899+0000 7fe5a27fc640 1 -- 172.21.15.17:0/1986985276 learned_addr learned my addr 172.21.15.17:0/1986985276 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:23:16.881 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.900+0000 7fe58b7fe640 10 client.?.objecter ms_handle_connect 0x7fe5a413f970 2024-03-29T11:23:16.881 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.900+0000 7fe58b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:16.881 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.900+0000 7fe58b7fe640 10 client.?.objecter ms_handle_connect 0x7fe5a40ae250 2024-03-29T11:23:16.881 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.900+0000 7fe58b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:16.881 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.900+0000 7fe58b7fe640 1 -- 172.21.15.17:0/1986985276 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1899281321 0 0) 0x7fe5a4145250 con 0x7fe5a413f970 2024-03-29T11:23:16.881 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.900+0000 7fe58b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:23:16.881 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.900+0000 7fe58b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:16.881 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.900+0000 7fe58b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:16.881 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.900+0000 7fe58b7fe640 10 monclient(hunting): my global_id is 4386 2024-03-29T11:23:16.881 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.900+0000 7fe58b7fe640 1 -- 172.21.15.17:0/1986985276 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe590004eb0 con 0x7fe5a413f970 2024-03-29T11:23:16.881 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.901+0000 7fe58b7fe640 1 -- 172.21.15.17:0/1986985276 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3459365856 0 0) 0x7fe5a406e7a0 con 0x7fe5a40ae250 2024-03-29T11:23:16.881 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.901+0000 7fe58b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:23:16.881 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.901+0000 7fe58b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:16.882 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.901+0000 7fe58b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:16.882 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.901+0000 7fe58b7fe640 10 monclient(hunting): my global_id is 4409 2024-03-29T11:23:16.882 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.901+0000 7fe58b7fe640 1 -- 172.21.15.17:0/1986985276 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5a4145250 con 0x7fe5a40ae250 2024-03-29T11:23:16.882 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.901+0000 7fe58b7fe640 1 -- 172.21.15.17:0/1986985276 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2073199472 0 0) 0x7fe5a4145250 con 0x7fe5a40ae250 2024-03-29T11:23:16.882 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.901+0000 7fe58b7fe640 1 -- 172.21.15.17:0/1986985276 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5a406e7a0 con 0x7fe5a40ae250 2024-03-29T11:23:16.882 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.901+0000 7fe58b7fe640 1 -- 172.21.15.17:0/1986985276 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fe594001870 con 0x7fe5a40ae250 2024-03-29T11:23:16.882 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.901+0000 7fe58b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:16.882 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.901+0000 7fe58b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:16.882 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.902+0000 7fe58b7fe640 10 monclient(hunting): dump: 2024-03-29T11:23:16.882 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:23:16.882 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:23:16.882 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:16.883 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:16.883 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:16.883 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:16.883 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:23:16.883 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:23:16.883 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:23:16.883 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:16.883 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.902+0000 7fe58b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:16.883 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.902+0000 7fe58b7fe640 1 -- 172.21.15.17:0/1986985276 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3323590191 0 0) 0x7fe590004eb0 con 0x7fe5a413f970 2024-03-29T11:23:16.883 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.902+0000 7fe58b7fe640 1 -- 172.21.15.17:0/1986985276 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5a4145250 con 0x7fe5a413f970 2024-03-29T11:23:16.883 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.902+0000 7fe58b7fe640 1 -- 172.21.15.17:0/1986985276 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fe59c001c70 con 0x7fe5a413f970 2024-03-29T11:23:16.883 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.902+0000 7fe58b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:16.883 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.902+0000 7fe58b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:16.883 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.902+0000 7fe58b7fe640 10 monclient(hunting): dump: 2024-03-29T11:23:16.883 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:23:16.884 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:23:16.884 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:16.884 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:16.884 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:16.884 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:16.884 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:23:16.884 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:23:16.884 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:23:16.884 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:16.884 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.902+0000 7fe58b7fe640 1 -- 172.21.15.17:0/1986985276 <== mon.2 v1:172.21.15.17:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3644097119 0 0) 0x7fe5a406e7a0 con 0x7fe5a40ae250 2024-03-29T11:23:16.884 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.902+0000 7fe58b7fe640 1 -- 172.21.15.17:0/1986985276 >> v1:172.21.15.94:6789/0 conn(0x7fe5a4142800 legacy=0x7fe5a413b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:16.884 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.902+0000 7fe58b7fe640 1 -- 172.21.15.17:0/1986985276 >> v1:172.21.15.17:6789/0 conn(0x7fe5a413f970 legacy=0x7fe5a413abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:16.884 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.902+0000 7fe58b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:23:16.884 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.902+0000 7fe58b7fe640 1 monclient: found mon.c 2024-03-29T11:23:16.884 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.902+0000 7fe58b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:16.885 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.902+0000 7fe58b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:23:16.885 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.902+0000 7fe58b7fe640 1 -- 172.21.15.17:0/1986985276 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a4144310 con 0x7fe5a40ae250 2024-03-29T11:23:16.885 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.902+0000 7fe58b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:23:16.885 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.902+0000 7fe58b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:23:16.885 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.902+0000 7fe58b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:16.885 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.902+0000 7fe5a93d6640 5 monclient: authenticate success, global_id 4409 2024-03-29T11:23:16.885 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.902+0000 7fe5a93d6640 10 monclient: _renew_subs 2024-03-29T11:23:16.885 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.902+0000 7fe5a93d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:23:16.885 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.902+0000 7fe5a93d6640 1 -- 172.21.15.17:0/1986985276 --> v1:172.21.15.17:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5a4144580 con 0x7fe5a40ae250 2024-03-29T11:23:16.885 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.902+0000 7fe5a93d6640 10 client.4409.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:16.885 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.902+0000 7fe5a93d6640 10 monclient: _renew_subs 2024-03-29T11:23:16.885 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.902+0000 7fe5a93d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:23:16.885 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.902+0000 7fe5a93d6640 1 -- 172.21.15.17:0/1986985276 --> v1:172.21.15.17:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5a41e7200 con 0x7fe5a40ae250 2024-03-29T11:23:16.885 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.902+0000 7fe5a93d6640 1 librados: init done 2024-03-29T11:23:16.885 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.902+0000 7fe58b7fe640 1 -- 172.21.15.17:0/1986985276 <== mon.2 v1:172.21.15.17:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe594001d60 con 0x7fe5a40ae250 2024-03-29T11:23:16.886 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.902+0000 7fe58b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:16.886 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.902+0000 7fe58b7fe640 1 -- 172.21.15.17:0/1986985276 <== mon.2 v1:172.21.15.17:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fe594003790 con 0x7fe5a40ae250 2024-03-29T11:23:16.886 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.902+0000 7fe58b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:16.886 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.902+0000 7fe58b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:16.886 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.902+0000 7fe58b7fe640 10 monclient: dump: 2024-03-29T11:23:16.886 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:23:16.886 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:23:16.886 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:16.886 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:16.886 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:16.886 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:16.886 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:23:16.886 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:23:16.886 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:23:16.886 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:16.888 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.903+0000 7fe5a0ff9640 4 set_mon_vals no callback set 2024-03-29T11:23:16.888 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.904+0000 7fe58b7fe640 1 -- 172.21.15.17:0/1986985276 <== mon.2 v1:172.21.15.17:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7fe5940038f0 con 0x7fe5a40ae250 2024-03-29T11:23:16.888 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.905+0000 7fe58b7fe640 1 -- 172.21.15.17:0/1986985276 <== mon.2 v1:172.21.15.17:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7fe594083760 con 0x7fe5a40ae250 2024-03-29T11:23:16.888 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.905+0000 7fe58b7fe640 10 client.4409.objecter ms_dispatch 0x7fe5a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:16.888 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.905+0000 7fe5a27fc640 10 monclient: get_auth_request con 0x7fe59006ddb0 auth_method 0 2024-03-29T11:23:16.888 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.905+0000 7fe58b7fe640 3 client.4409.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:16.888 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.905+0000 7fe58b7fe640 3 client.4409.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:16.888 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.905+0000 7fe58b7fe640 20 client.4409.objecter dump_active .. 0 homeless 2024-03-29T11:23:16.888 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.905+0000 7fe58b7fe640 10 client.4409.objecter ms_handle_connect 0x7fe59006ddb0 2024-03-29T11:23:16.888 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.905+0000 7fe5a93d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:16.889 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.906+0000 7fe5a93d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:16.889 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.906+0000 7fe5a93d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:23:16.889 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.906+0000 7fe5a93d6640 1 -- 172.21.15.17:0/1986985276 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe584005080 con 0x7fe5a40ae250 2024-03-29T11:23:16.892 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.914+0000 7fe58b7fe640 1 -- 172.21.15.17:0/1986985276 <== mon.2 v1:172.21.15.17:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe59404ee70 con 0x7fe5a40ae250 2024-03-29T11:23:16.892 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.914+0000 7fe58b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:16.892 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:16.914+0000 7fe58b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:17.219 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:17.240+0000 7fe5a93d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:17.219 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:17.240+0000 7fe5a93d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:17.219 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:17.240+0000 7fe5a93d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:23:17.219 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:17.240+0000 7fe5a93d6640 1 -- 172.21.15.17:0/1986985276 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe584005330 con 0x7fe5a40ae250 2024-03-29T11:23:17.222 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:17.243+0000 7fe58b7fe640 1 -- 172.21.15.17:0/1986985276 <== mon.2 v1:172.21.15.17:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7fe594050f70 con 0x7fe5a40ae250 2024-03-29T11:23:17.222 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:17.243+0000 7fe58b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:17.222 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:17.243+0000 7fe58b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:17.223 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:23:17.223 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:17.225 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:17.247+0000 7fe5a93d6640 10 librados: watch_flush enter 2024-03-29T11:23:17.225 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:17.247+0000 7fe5a93d6640 10 librados: watch_flush exit 2024-03-29T11:23:17.226 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:17.247+0000 7fe5a93d6640 20 client.4409.objecter shutdown clearing up homeless session... 2024-03-29T11:23:17.226 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:17.247+0000 7fe5a93d6640 10 client.4409.objecter successfully canceled tick 2024-03-29T11:23:17.226 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:17.247+0000 7fe5a93d6640 1 -- 172.21.15.17:0/1986985276 >> v1:172.21.15.17:6816/3236185052 conn(0x7fe59006ddb0 legacy=0x7fe590070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:17.226 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:17.247+0000 7fe5a93d6640 10 monclient: shutdown 2024-03-29T11:23:17.227 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:17.247+0000 7fe5a93d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:17.227 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:17.247+0000 7fe5a93d6640 1 -- 172.21.15.17:0/1986985276 >> v1:172.21.15.17:6790/0 conn(0x7fe5a40ae250 legacy=0x7fe5a41449d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:17.229 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:17.250+0000 7fe5a93d6640 1 -- 172.21.15.17:0/1986985276 shutdown_connections 2024-03-29T11:23:17.229 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:17.250+0000 7fe5a93d6640 1 -- 172.21.15.17:0/1986985276 >> 172.21.15.17:0/1986985276 conn(0x7fe5a40b35e0 msgr2=0x7fe5a40aa090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:17.229 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:17.251+0000 7fe5a93d6640 1 -- 172.21.15.17:0/1986985276 shutdown_connections 2024-03-29T11:23:17.229 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:17.251+0000 7fe5a93d6640 1 -- 172.21.15.17:0/1986985276 wait complete. 2024-03-29T11:23:17.230 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:17.252+0000 7fe5a93d6640 1 librados: shutdown 2024-03-29T11:23:17.247 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:24.054 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:24.210 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.231+0000 7f3a53fff640 10 monclient: get_monmap_and_config 2024-03-29T11:23:24.211 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.231+0000 7f3a53fff640 10 monclient: build_initial_monmap 2024-03-29T11:23:24.211 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.232+0000 7f3a53fff640 10 monclient: monmap: 2024-03-29T11:23:24.211 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:23:24.211 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:23:24.211 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:23:24.232965+0000 2024-03-29T11:23:24.211 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:23:24.232965+0000 2024-03-29T11:23:24.211 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:24.211 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:24.211 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:23:24.211 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:23:24.212 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:23:24.212 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:24.212 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.233+0000 7f3a53fff640 1 Processor -- start 2024-03-29T11:23:24.212 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.233+0000 7f3a53fff640 1 -- start start 2024-03-29T11:23:24.212 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.233+0000 7f3a53fff640 10 monclient: init 2024-03-29T11:23:24.213 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.234+0000 7f3a53fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:24.213 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.234+0000 7f3a53fff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:23:24.213 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.234+0000 7f3a53fff640 10 monclient(hunting): picked mon.noname-c con 0x7f3a4c006a70 addr v1:172.21.15.94:6789/0 2024-03-29T11:23:24.213 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.234+0000 7f3a53fff640 10 monclient(hunting): picked mon.noname-a con 0x7f3a4c02b100 addr v1:172.21.15.17:6789/0 2024-03-29T11:23:24.213 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.234+0000 7f3a53fff640 10 monclient(hunting): picked mon.noname-b con 0x7f3a4c029c30 addr v1:172.21.15.17:6790/0 2024-03-29T11:23:24.213 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.234+0000 7f3a53fff640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a4c02a270 con 0x7f3a4c02b100 2024-03-29T11:23:24.213 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.234+0000 7f3a53fff640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a4c02c440 con 0x7f3a4c006a70 2024-03-29T11:23:24.213 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.234+0000 7f3a53fff640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a4c0c9440 con 0x7f3a4c029c30 2024-03-29T11:23:24.214 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.234+0000 7f3a53fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:24.214 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.234+0000 7f3a53fff640 10 monclient(hunting): authenticate will time out at 1110.439247s 2024-03-29T11:23:24.214 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.234+0000 7f3a52ffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f3a4c006a70 0x7f3a4c0c8d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:42566/0 (socket says 172.21.15.17:42566) 2024-03-29T11:23:24.214 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.234+0000 7f3a52ffd640 1 -- 172.21.15.17:0/3844212395 learned_addr learned my addr 172.21.15.17:0/3844212395 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:23:24.218 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.235+0000 7f3a51ffb640 1 -- 172.21.15.17:0/3844212395 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1300160488 0 0) 0x7f3a4c02c440 con 0x7f3a4c006a70 2024-03-29T11:23:24.218 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.235+0000 7f3a51ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:24.218 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.235+0000 7f3a51ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:24.218 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.235+0000 7f3a51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:24.218 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.235+0000 7f3a51ffb640 10 monclient(hunting): my global_id is 4399 2024-03-29T11:23:24.218 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.235+0000 7f3a51ffb640 1 -- 172.21.15.17:0/3844212395 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a3c003520 con 0x7f3a4c006a70 2024-03-29T11:23:24.218 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.235+0000 7f3a51ffb640 1 -- 172.21.15.17:0/3844212395 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2201373931 0 0) 0x7f3a4c0c9440 con 0x7f3a4c029c30 2024-03-29T11:23:24.218 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.235+0000 7f3a51ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:24.218 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.235+0000 7f3a51ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:24.218 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.235+0000 7f3a51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:24.219 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.235+0000 7f3a51ffb640 10 monclient(hunting): my global_id is 4412 2024-03-29T11:23:24.219 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.235+0000 7f3a51ffb640 1 -- 172.21.15.17:0/3844212395 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a4c02c440 con 0x7f3a4c029c30 2024-03-29T11:23:24.219 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.235+0000 7f3a51ffb640 1 -- 172.21.15.17:0/3844212395 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2169729843 0 0) 0x7f3a4c02a270 con 0x7f3a4c02b100 2024-03-29T11:23:24.219 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.235+0000 7f3a51ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:24.219 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.235+0000 7f3a51ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:24.219 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.235+0000 7f3a51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:24.219 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.235+0000 7f3a51ffb640 10 monclient(hunting): my global_id is 4389 2024-03-29T11:23:24.219 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.235+0000 7f3a51ffb640 1 -- 172.21.15.17:0/3844212395 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a4c0c9440 con 0x7f3a4c02b100 2024-03-29T11:23:24.219 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.236+0000 7f3a51ffb640 1 -- 172.21.15.17:0/3844212395 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1218373239 0 0) 0x7f3a3c003520 con 0x7f3a4c006a70 2024-03-29T11:23:24.219 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.236+0000 7f3a51ffb640 1 -- 172.21.15.17:0/3844212395 >> v1:172.21.15.17:6790/0 conn(0x7f3a4c029c30 legacy=0x7f3a4c0d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:24.219 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.236+0000 7f3a51ffb640 1 -- 172.21.15.17:0/3844212395 >> v1:172.21.15.17:6789/0 conn(0x7f3a4c02b100 legacy=0x7f3a4c0cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:24.219 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.236+0000 7f3a51ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:23:24.219 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.236+0000 7f3a51ffb640 1 monclient: found mon.noname-c 2024-03-29T11:23:24.219 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.236+0000 7f3a51ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:24.220 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.236+0000 7f3a51ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6789/0 2024-03-29T11:23:24.220 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.236+0000 7f3a51ffb640 1 -- 172.21.15.17:0/3844212395 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a4c0c95d0 con 0x7f3a4c006a70 2024-03-29T11:23:24.220 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.236+0000 7f3a51ffb640 10 monclient: _finish_auth 0 2024-03-29T11:23:24.220 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.236+0000 7f3a51ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:23:24.220 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.236+0000 7f3a51ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:24.220 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.236+0000 7f3a51ffb640 1 -- 172.21.15.17:0/3844212395 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f3a44001690 con 0x7f3a4c006a70 2024-03-29T11:23:24.220 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.236+0000 7f3a51ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:24.220 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.236+0000 7f3a51ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:24.220 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.236+0000 7f3a51ffb640 10 monclient: dump: 2024-03-29T11:23:24.220 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:23:24.220 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:23:24.220 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:24.221 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:24.221 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:24.221 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:24.221 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:23:24.221 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:23:24.221 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:23:24.221 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:24.221 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.236+0000 7f3a53fff640 5 monclient: authenticate success, global_id 4399 2024-03-29T11:23:24.221 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.236+0000 7f3a53fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:24.221 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.237+0000 7f3a51ffb640 1 -- 172.21.15.17:0/3844212395 <== mon.1 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a440017f0 con 0x7f3a4c006a70 2024-03-29T11:23:24.221 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.237+0000 7f3a51ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:24.222 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.237+0000 7f3a53fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:24.222 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.237+0000 7f3a53fff640 4 set_mon_vals no callback set 2024-03-29T11:23:24.222 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.238+0000 7f3a51ffb640 1 -- 172.21.15.17:0/3844212395 <== mon.1 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f3a44001c00 con 0x7f3a4c006a70 2024-03-29T11:23:24.222 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.238+0000 7f3a51ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:24.222 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.238+0000 7f3a51ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:24.222 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.238+0000 7f3a51ffb640 10 monclient: dump: 2024-03-29T11:23:24.222 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:23:24.222 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:23:24.222 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:24.223 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:24.223 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:24.223 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:24.223 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:23:24.223 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:23:24.223 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:23:24.223 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:24.223 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.238+0000 7f3a53fff640 10 monclient: shutdown 2024-03-29T11:23:24.223 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.238+0000 7f3a53fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:24.223 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.238+0000 7f3a53fff640 1 -- 172.21.15.17:0/3844212395 >> v1:172.21.15.94:6789/0 conn(0x7f3a4c006a70 legacy=0x7f3a4c0c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:24.223 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.238+0000 7f3a53fff640 1 -- 172.21.15.17:0/3844212395 shutdown_connections 2024-03-29T11:23:24.223 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.238+0000 7f3a53fff640 1 -- 172.21.15.17:0/3844212395 >> 172.21.15.17:0/3844212395 conn(0x7f3a4c0058f0 msgr2=0x7f3a4c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:24.223 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.238+0000 7f3a53fff640 1 -- 172.21.15.17:0/3844212395 shutdown_connections 2024-03-29T11:23:24.223 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.238+0000 7f3a53fff640 1 -- 172.21.15.17:0/3844212395 wait complete. 2024-03-29T11:23:24.230 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.239+0000 7f3a53fff640 10 monclient: build_initial_monmap 2024-03-29T11:23:24.230 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.239+0000 7f3a53fff640 10 monclient: monmap: 2024-03-29T11:23:24.230 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:23:24.230 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:24.230 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:23:24.230 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:23:24.231 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:24.231 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:24.231 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:23:24.231 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:23:24.231 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:23:24.231 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:24.231 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.240+0000 7f3a53fff640 1 librados: starting msgr at 2024-03-29T11:23:24.231 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.240+0000 7f3a53fff640 1 librados: starting objecter 2024-03-29T11:23:24.231 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.240+0000 7f3a53fff640 1 Processor -- start 2024-03-29T11:23:24.231 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.240+0000 7f3a53fff640 1 -- start start 2024-03-29T11:23:24.231 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.240+0000 7f3a53fff640 1 librados: setting wanted keys 2024-03-29T11:23:24.231 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.240+0000 7f3a53fff640 1 librados: calling monclient init 2024-03-29T11:23:24.231 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.240+0000 7f3a53fff640 10 monclient: init 2024-03-29T11:23:24.231 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.241+0000 7f3a53fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:24.232 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.241+0000 7f3a53fff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:23:24.232 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.241+0000 7f3a53fff640 10 monclient(hunting): picked mon.noname-b con 0x7f3a4c006a70 addr v1:172.21.15.94:6789/0 2024-03-29T11:23:24.232 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.242+0000 7f3a53fff640 10 monclient(hunting): picked mon.noname-a con 0x7f3a4c029c30 addr v1:172.21.15.17:6789/0 2024-03-29T11:23:24.232 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.242+0000 7f3a52ffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f3a4c006a70 0x7f3a4c158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:44618/0 (socket says 172.21.15.17:44618) 2024-03-29T11:23:24.232 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.242+0000 7f3a53fff640 10 monclient(hunting): picked mon.noname-c con 0x7f3a4c0c2450 addr v1:172.21.15.17:6790/0 2024-03-29T11:23:24.232 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.242+0000 7f3a52ffd640 1 -- 172.21.15.17:0/2192319002 learned_addr learned my addr 172.21.15.17:0/2192319002 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:23:24.232 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.242+0000 7f3a53fff640 1 -- 172.21.15.17:0/2192319002 --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a4c158a30 con 0x7f3a4c029c30 2024-03-29T11:23:24.232 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.242+0000 7f3a53fff640 1 -- 172.21.15.17:0/2192319002 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a4c0c2230 con 0x7f3a4c006a70 2024-03-29T11:23:24.232 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.242+0000 7f3a53fff640 1 -- 172.21.15.17:0/2192319002 --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a4c1576a0 con 0x7f3a4c0c2450 2024-03-29T11:23:24.232 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.242+0000 7f3a53fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:24.232 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.242+0000 7f3a53fff640 10 monclient(hunting): authenticate will time out at 1110.447256s 2024-03-29T11:23:24.232 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.242+0000 7f3a3b7fe640 10 client.?.objecter ms_handle_connect 0x7f3a4c006a70 2024-03-29T11:23:24.232 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.242+0000 7f3a3b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:24.232 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.242+0000 7f3a3b7fe640 10 client.?.objecter ms_handle_connect 0x7f3a4c029c30 2024-03-29T11:23:24.233 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.242+0000 7f3a3b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:24.233 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.242+0000 7f3a3b7fe640 10 client.?.objecter ms_handle_connect 0x7f3a4c0c2450 2024-03-29T11:23:24.233 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.242+0000 7f3a3b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:24.233 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.243+0000 7f3a3b7fe640 1 -- 172.21.15.17:0/2192319002 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1681161835 0 0) 0x7f3a4c0c2230 con 0x7f3a4c006a70 2024-03-29T11:23:24.233 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.243+0000 7f3a3b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:23:24.233 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.243+0000 7f3a3b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:24.233 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.243+0000 7f3a3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:24.233 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.243+0000 7f3a3b7fe640 10 monclient(hunting): my global_id is 4402 2024-03-29T11:23:24.233 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.243+0000 7f3a3b7fe640 1 -- 172.21.15.17:0/2192319002 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a40004eb0 con 0x7f3a4c006a70 2024-03-29T11:23:24.233 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.243+0000 7f3a3b7fe640 1 -- 172.21.15.17:0/2192319002 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2960724285 0 0) 0x7f3a4c158a30 con 0x7f3a4c029c30 2024-03-29T11:23:24.233 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.243+0000 7f3a3b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:23:24.233 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.243+0000 7f3a3b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:24.234 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.243+0000 7f3a3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:24.234 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.243+0000 7f3a3b7fe640 10 monclient(hunting): my global_id is 4392 2024-03-29T11:23:24.234 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.243+0000 7f3a3b7fe640 1 -- 172.21.15.17:0/2192319002 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a4c0c2230 con 0x7f3a4c029c30 2024-03-29T11:23:24.234 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.243+0000 7f3a3b7fe640 1 -- 172.21.15.17:0/2192319002 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1741042813 0 0) 0x7f3a40004eb0 con 0x7f3a4c006a70 2024-03-29T11:23:24.234 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.243+0000 7f3a3b7fe640 1 -- 172.21.15.17:0/2192319002 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3a4c158a30 con 0x7f3a4c006a70 2024-03-29T11:23:24.234 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.243+0000 7f3a3b7fe640 1 -- 172.21.15.17:0/2192319002 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f3a44001bc0 con 0x7f3a4c006a70 2024-03-29T11:23:24.234 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.243+0000 7f3a3b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:24.234 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.243+0000 7f3a3b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:24.234 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.243+0000 7f3a3b7fe640 10 monclient(hunting): dump: 2024-03-29T11:23:24.234 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:23:24.234 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:23:24.234 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:24.234 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:24.234 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:24.234 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:24.235 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:23:24.235 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:23:24.235 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:23:24.235 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:24.235 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.243+0000 7f3a3b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:24.235 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.244+0000 7f3a3b7fe640 1 -- 172.21.15.17:0/2192319002 <== mon.1 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 287986537 0 0) 0x7f3a4c158a30 con 0x7f3a4c006a70 2024-03-29T11:23:24.235 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.244+0000 7f3a3b7fe640 1 -- 172.21.15.17:0/2192319002 >> v1:172.21.15.17:6790/0 conn(0x7f3a4c0c2450 legacy=0x7f3a4c16b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:24.235 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.244+0000 7f3a3b7fe640 1 -- 172.21.15.17:0/2192319002 >> v1:172.21.15.17:6789/0 conn(0x7f3a4c029c30 legacy=0x7f3a4c161240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:24.235 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.244+0000 7f3a3b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:23:24.235 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.244+0000 7f3a3b7fe640 1 monclient: found mon.b 2024-03-29T11:23:24.235 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.244+0000 7f3a3b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:24.235 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.244+0000 7f3a3b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:23:24.235 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.244+0000 7f3a3b7fe640 1 -- 172.21.15.17:0/2192319002 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a4c157980 con 0x7f3a4c006a70 2024-03-29T11:23:24.235 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.244+0000 7f3a3b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:23:24.235 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.244+0000 7f3a3b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:23:24.236 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.244+0000 7f3a3b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:24.236 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.244+0000 7f3a53fff640 5 monclient: authenticate success, global_id 4402 2024-03-29T11:23:24.236 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.244+0000 7f3a53fff640 10 monclient: _renew_subs 2024-03-29T11:23:24.236 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.244+0000 7f3a53fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:23:24.236 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.244+0000 7f3a53fff640 1 -- 172.21.15.17:0/2192319002 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a4c161950 con 0x7f3a4c006a70 2024-03-29T11:23:24.236 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.244+0000 7f3a53fff640 10 client.4402.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:24.236 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.244+0000 7f3a53fff640 10 monclient: _renew_subs 2024-03-29T11:23:24.236 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.244+0000 7f3a53fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:23:24.236 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.244+0000 7f3a53fff640 1 -- 172.21.15.17:0/2192319002 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a4c161e30 con 0x7f3a4c006a70 2024-03-29T11:23:24.236 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.244+0000 7f3a53fff640 1 librados: init done 2024-03-29T11:23:24.236 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.245+0000 7f3a3b7fe640 1 -- 172.21.15.17:0/2192319002 <== mon.1 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a44004a90 con 0x7f3a4c006a70 2024-03-29T11:23:24.236 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.245+0000 7f3a3b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:24.236 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.245+0000 7f3a3b7fe640 1 -- 172.21.15.17:0/2192319002 <== mon.1 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f3a44004da0 con 0x7f3a4c006a70 2024-03-29T11:23:24.236 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.245+0000 7f3a3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:24.236 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.245+0000 7f3a517fa640 4 set_mon_vals no callback set 2024-03-29T11:23:24.237 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.245+0000 7f3a3b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:24.237 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.245+0000 7f3a3b7fe640 10 monclient: dump: 2024-03-29T11:23:24.237 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:23:24.237 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:23:24.237 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:24.237 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:24.237 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:24.237 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:24.237 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:23:24.237 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:23:24.237 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:23:24.237 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:24.237 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.245+0000 7f3a53fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:24.237 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.246+0000 7f3a53fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:24.238 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.246+0000 7f3a53fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:23:24.238 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.246+0000 7f3a53fff640 1 -- 172.21.15.17:0/2192319002 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a4c004cd0 con 0x7f3a4c006a70 2024-03-29T11:23:24.238 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.247+0000 7f3a3b7fe640 1 -- 172.21.15.17:0/2192319002 <== mon.1 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f3a4401ad70 con 0x7f3a4c006a70 2024-03-29T11:23:24.238 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.248+0000 7f3a3b7fe640 1 -- 172.21.15.17:0/2192319002 <== mon.1 v1:172.21.15.94:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f3a44084bc0 con 0x7f3a4c006a70 2024-03-29T11:23:24.238 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.248+0000 7f3a3b7fe640 10 client.4402.objecter ms_dispatch 0x7f3a54000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:24.238 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.248+0000 7f3a3b7fe640 3 client.4402.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:24.238 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.248+0000 7f3a3b7fe640 3 client.4402.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:24.238 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.248+0000 7f3a527fc640 10 monclient: get_auth_request con 0x7f3a4006db90 auth_method 0 2024-03-29T11:23:24.238 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.248+0000 7f3a3b7fe640 20 client.4402.objecter dump_active .. 0 homeless 2024-03-29T11:23:24.238 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.249+0000 7f3a3b7fe640 10 client.4402.objecter ms_handle_connect 0x7f3a4006db90 2024-03-29T11:23:24.239 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.254+0000 7f3a3b7fe640 1 -- 172.21.15.17:0/2192319002 <== mon.1 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3a440504a0 con 0x7f3a4c006a70 2024-03-29T11:23:24.239 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.254+0000 7f3a3b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:24.239 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.254+0000 7f3a3b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:24.619 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.640+0000 7f3a53fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:24.619 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.640+0000 7f3a53fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:24.619 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.640+0000 7f3a53fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:23:24.619 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.640+0000 7f3a53fff640 1 -- 172.21.15.17:0/2192319002 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3a4c0008d0 con 0x7f3a4c006a70 2024-03-29T11:23:24.620 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.641+0000 7f3a3b7fe640 1 -- 172.21.15.17:0/2192319002 <== mon.1 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f3a440523d0 con 0x7f3a4c006a70 2024-03-29T11:23:24.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.641+0000 7f3a3b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:24.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.641+0000 7f3a3b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:24.621 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:23:24.621 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:24.624 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.646+0000 7f3a53fff640 10 librados: watch_flush enter 2024-03-29T11:23:24.625 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.646+0000 7f3a53fff640 10 librados: watch_flush exit 2024-03-29T11:23:24.625 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.646+0000 7f3a53fff640 20 client.4402.objecter shutdown clearing up homeless session... 2024-03-29T11:23:24.625 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.646+0000 7f3a53fff640 10 client.4402.objecter successfully canceled tick 2024-03-29T11:23:24.626 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.646+0000 7f3a53fff640 1 -- 172.21.15.17:0/2192319002 >> v1:172.21.15.17:6816/3236185052 conn(0x7f3a4006db90 legacy=0x7f3a40070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:24.626 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.646+0000 7f3a53fff640 10 monclient: shutdown 2024-03-29T11:23:24.626 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.646+0000 7f3a53fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:24.626 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.646+0000 7f3a53fff640 1 -- 172.21.15.17:0/2192319002 >> v1:172.21.15.94:6789/0 conn(0x7f3a4c006a70 legacy=0x7f3a4c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:24.626 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.646+0000 7f3a53fff640 1 -- 172.21.15.17:0/2192319002 shutdown_connections 2024-03-29T11:23:24.626 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.647+0000 7f3a53fff640 1 -- 172.21.15.17:0/2192319002 >> 172.21.15.17:0/2192319002 conn(0x7f3a4c0058f0 msgr2=0x7f3a4c0376b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:24.626 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.647+0000 7f3a53fff640 1 -- 172.21.15.17:0/2192319002 shutdown_connections 2024-03-29T11:23:24.627 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.648+0000 7f3a53fff640 1 -- 172.21.15.17:0/2192319002 wait complete. 2024-03-29T11:23:24.628 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:24.649+0000 7f3a53fff640 1 librados: shutdown 2024-03-29T11:23:24.645 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:31.452 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:31.600 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.621+0000 7fd17b59e640 10 monclient: get_monmap_and_config 2024-03-29T11:23:31.600 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.621+0000 7fd17b59e640 10 monclient: build_initial_monmap 2024-03-29T11:23:31.600 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.621+0000 7fd17b59e640 10 monclient: monmap: 2024-03-29T11:23:31.600 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:23:31.601 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:23:31.601 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:23:31.622121+0000 2024-03-29T11:23:31.601 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:23:31.622121+0000 2024-03-29T11:23:31.601 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:31.601 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:31.601 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:23:31.601 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:23:31.601 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:23:31.601 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:31.602 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.621+0000 7fd17b59e640 1 Processor -- start 2024-03-29T11:23:31.602 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.621+0000 7fd17b59e640 1 -- start start 2024-03-29T11:23:31.602 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.621+0000 7fd17b59e640 10 monclient: init 2024-03-29T11:23:31.602 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.622+0000 7fd17b59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:31.602 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.622+0000 7fd17b59e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:23:31.603 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.622+0000 7fd17b59e640 10 monclient(hunting): picked mon.noname-a con 0x7fd174038c60 addr v1:172.21.15.17:6789/0 2024-03-29T11:23:31.603 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.622+0000 7fd17b59e640 10 monclient(hunting): picked mon.noname-c con 0x7fd1740324b0 addr v1:172.21.15.94:6789/0 2024-03-29T11:23:31.603 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.622+0000 7fd17b59e640 10 monclient(hunting): picked mon.noname-b con 0x7fd1740d11e0 addr v1:172.21.15.17:6790/0 2024-03-29T11:23:31.603 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.622+0000 7fd17b59e640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1740caf80 con 0x7fd174038c60 2024-03-29T11:23:31.603 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.622+0000 7fd17b59e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd174039850 con 0x7fd1740324b0 2024-03-29T11:23:31.603 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.622+0000 7fd17b59e640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1740c78d0 con 0x7fd1740d11e0 2024-03-29T11:23:31.603 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.622+0000 7fd17b59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:31.603 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.622+0000 7fd17b59e640 10 monclient(hunting): authenticate will time out at 1117.827848s 2024-03-29T11:23:31.603 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.623+0000 7fd17ad9d640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7fd1740d11e0 0x7fd1740d35d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:38870/0 (socket says 172.21.15.17:38870) 2024-03-29T11:23:31.603 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.623+0000 7fd17ad9d640 1 -- 172.21.15.17:0/1634494138 learned_addr learned my addr 172.21.15.17:0/1634494138 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:23:31.604 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.623+0000 7fd17959a640 1 -- 172.21.15.17:0/1634494138 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 955919642 0 0) 0x7fd174039850 con 0x7fd1740324b0 2024-03-29T11:23:31.604 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.623+0000 7fd17959a640 10 monclient(hunting): _negotiate 2024-03-29T11:23:31.605 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.623+0000 7fd17959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:31.605 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.623+0000 7fd17959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:31.605 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.623+0000 7fd17959a640 10 monclient(hunting): my global_id is 4405 2024-03-29T11:23:31.605 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.624+0000 7fd17959a640 1 -- 172.21.15.17:0/1634494138 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd164003520 con 0x7fd1740324b0 2024-03-29T11:23:31.605 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.624+0000 7fd17959a640 1 -- 172.21.15.17:0/1634494138 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3720929837 0 0) 0x7fd1740c78d0 con 0x7fd1740d11e0 2024-03-29T11:23:31.605 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.624+0000 7fd17959a640 10 monclient(hunting): _negotiate 2024-03-29T11:23:31.605 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.624+0000 7fd17959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:31.605 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.624+0000 7fd17959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:31.605 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.624+0000 7fd17959a640 10 monclient(hunting): my global_id is 4418 2024-03-29T11:23:31.605 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.624+0000 7fd17959a640 1 -- 172.21.15.17:0/1634494138 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1640048a0 con 0x7fd1740d11e0 2024-03-29T11:23:31.605 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.624+0000 7fd17959a640 1 -- 172.21.15.17:0/1634494138 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3313262526 0 0) 0x7fd164003520 con 0x7fd1740324b0 2024-03-29T11:23:31.605 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.624+0000 7fd17959a640 1 -- 172.21.15.17:0/1634494138 >> v1:172.21.15.17:6790/0 conn(0x7fd1740d11e0 legacy=0x7fd1740d35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:31.605 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.624+0000 7fd17959a640 1 -- 172.21.15.17:0/1634494138 >> v1:172.21.15.17:6789/0 conn(0x7fd174038c60 legacy=0x7fd1740c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:31.605 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.624+0000 7fd17959a640 10 monclient: _finish_hunting 0 2024-03-29T11:23:31.606 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.624+0000 7fd17959a640 1 monclient: found mon.noname-c 2024-03-29T11:23:31.606 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.624+0000 7fd17959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:31.606 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.624+0000 7fd17959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6789/0 2024-03-29T11:23:31.606 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.624+0000 7fd17959a640 1 -- 172.21.15.17:0/1634494138 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1740c7a60 con 0x7fd1740324b0 2024-03-29T11:23:31.606 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.624+0000 7fd17959a640 10 monclient: _finish_auth 0 2024-03-29T11:23:31.606 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.624+0000 7fd17959a640 10 monclient: _check_auth_tickets 2024-03-29T11:23:31.606 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.624+0000 7fd17959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:31.606 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.624+0000 7fd17959a640 1 -- 172.21.15.17:0/1634494138 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fd170001670 con 0x7fd1740324b0 2024-03-29T11:23:31.606 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.624+0000 7fd17959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:31.606 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.624+0000 7fd17959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:31.606 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.624+0000 7fd17959a640 10 monclient: dump: 2024-03-29T11:23:31.606 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:23:31.606 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:23:31.607 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:31.607 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:31.607 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:31.607 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:31.607 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:23:31.607 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:23:31.607 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:23:31.607 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:31.607 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.624+0000 7fd17b59e640 5 monclient: authenticate success, global_id 4405 2024-03-29T11:23:31.607 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.624+0000 7fd17b59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:31.608 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.626+0000 7fd17959a640 1 -- 172.21.15.17:0/1634494138 <== mon.1 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd170001840 con 0x7fd1740324b0 2024-03-29T11:23:31.608 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.626+0000 7fd17959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:31.609 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.626+0000 7fd17959a640 1 -- 172.21.15.17:0/1634494138 <== mon.1 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fd170001670 con 0x7fd1740324b0 2024-03-29T11:23:31.609 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.626+0000 7fd17959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:31.609 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.626+0000 7fd17959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:31.609 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.626+0000 7fd17959a640 10 monclient: dump: 2024-03-29T11:23:31.609 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:23:31.609 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:23:31.609 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:31.609 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:31.609 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:31.609 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:31.609 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:23:31.609 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:23:31.609 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:23:31.609 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:31.609 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.626+0000 7fd17b59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:31.610 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.626+0000 7fd17b59e640 4 set_mon_vals no callback set 2024-03-29T11:23:31.610 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.627+0000 7fd17b59e640 10 monclient: shutdown 2024-03-29T11:23:31.610 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.627+0000 7fd17b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:31.610 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.627+0000 7fd17b59e640 1 -- 172.21.15.17:0/1634494138 >> v1:172.21.15.94:6789/0 conn(0x7fd1740324b0 legacy=0x7fd1740cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:31.610 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.627+0000 7fd17b59e640 1 -- 172.21.15.17:0/1634494138 shutdown_connections 2024-03-29T11:23:31.610 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.627+0000 7fd17b59e640 1 -- 172.21.15.17:0/1634494138 >> 172.21.15.17:0/1634494138 conn(0x7fd174005c20 msgr2=0x7fd174006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:31.610 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.628+0000 7fd17b59e640 1 -- 172.21.15.17:0/1634494138 shutdown_connections 2024-03-29T11:23:31.610 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.628+0000 7fd17b59e640 1 -- 172.21.15.17:0/1634494138 wait complete. 2024-03-29T11:23:31.610 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.629+0000 7fd17b59e640 10 monclient: build_initial_monmap 2024-03-29T11:23:31.610 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.629+0000 7fd17b59e640 10 monclient: monmap: 2024-03-29T11:23:31.610 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:23:31.610 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:31.610 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:23:31.611 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:23:31.611 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:31.611 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:31.611 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:23:31.611 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:23:31.611 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:23:31.611 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:31.615 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.630+0000 7fd17b59e640 1 librados: starting msgr at 2024-03-29T11:23:31.615 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.630+0000 7fd17b59e640 1 librados: starting objecter 2024-03-29T11:23:31.615 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.630+0000 7fd17b59e640 1 Processor -- start 2024-03-29T11:23:31.615 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.630+0000 7fd17b59e640 1 -- start start 2024-03-29T11:23:31.615 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.630+0000 7fd17b59e640 1 librados: setting wanted keys 2024-03-29T11:23:31.615 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.630+0000 7fd17b59e640 1 librados: calling monclient init 2024-03-29T11:23:31.615 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.630+0000 7fd17b59e640 10 monclient: init 2024-03-29T11:23:31.615 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.631+0000 7fd17b59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:31.615 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.631+0000 7fd17b59e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:23:31.615 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.631+0000 7fd17b59e640 10 monclient(hunting): picked mon.noname-b con 0x7fd1740324b0 addr v1:172.21.15.94:6789/0 2024-03-29T11:23:31.616 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.631+0000 7fd17b59e640 10 monclient(hunting): picked mon.noname-c con 0x7fd174038c60 addr v1:172.21.15.17:6790/0 2024-03-29T11:23:31.616 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.632+0000 7fd179d9b640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7fd174038c60 0x7fd174036c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:40262/0 (socket says 172.21.15.17:40262) 2024-03-29T11:23:31.616 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.632+0000 7fd17a59c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fd1740324b0 0x7fd1740c4fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:46090/0 (socket says 172.21.15.17:46090) 2024-03-29T11:23:31.616 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.632+0000 7fd17ad9d640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fd1740d11e0 0x7fd1740348b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:40262/0 (socket says 172.21.15.17:40262) 2024-03-29T11:23:31.616 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.632+0000 7fd17b59e640 10 monclient(hunting): picked mon.noname-a con 0x7fd1740d11e0 addr v1:172.21.15.17:6789/0 2024-03-29T11:23:31.616 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.632+0000 7fd17b59e640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd174036500 con 0x7fd1740d11e0 2024-03-29T11:23:31.616 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.632+0000 7fd17b59e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd174036730 con 0x7fd1740324b0 2024-03-29T11:23:31.616 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.632+0000 7fd17b59e640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd17416e460 con 0x7fd174038c60 2024-03-29T11:23:31.616 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.632+0000 7fd17b59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:31.616 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.632+0000 7fd17b59e640 10 monclient(hunting): authenticate will time out at 1117.837216s 2024-03-29T11:23:31.616 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.632+0000 7fd179d9b640 1 -- 172.21.15.17:0/584204593 learned_addr learned my addr 172.21.15.17:0/584204593 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:23:31.616 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.632+0000 7fd16affd640 10 client.?.objecter ms_handle_connect 0x7fd1740d11e0 2024-03-29T11:23:31.616 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.632+0000 7fd16affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:31.616 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.632+0000 7fd16affd640 10 client.?.objecter ms_handle_connect 0x7fd174038c60 2024-03-29T11:23:31.616 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.632+0000 7fd16affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:31.617 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.632+0000 7fd16affd640 1 -- 172.21.15.17:0/584204593 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1834801262 0 0) 0x7fd174036500 con 0x7fd1740d11e0 2024-03-29T11:23:31.617 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.632+0000 7fd16affd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:31.617 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.632+0000 7fd16affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:31.617 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.632+0000 7fd16affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:31.617 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.632+0000 7fd16affd640 10 monclient(hunting): my global_id is 4398 2024-03-29T11:23:31.617 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.632+0000 7fd16affd640 1 -- 172.21.15.17:0/584204593 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd15c004eb0 con 0x7fd1740d11e0 2024-03-29T11:23:31.617 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.632+0000 7fd16affd640 10 client.?.objecter ms_handle_connect 0x7fd1740324b0 2024-03-29T11:23:31.617 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.632+0000 7fd16affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:31.617 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.632+0000 7fd16affd640 1 -- 172.21.15.17:0/584204593 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 848417257 0 0) 0x7fd17416e460 con 0x7fd174038c60 2024-03-29T11:23:31.617 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.632+0000 7fd16affd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:31.617 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.632+0000 7fd16affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:31.617 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.632+0000 7fd16affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:31.617 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.632+0000 7fd16affd640 10 monclient(hunting): my global_id is 4421 2024-03-29T11:23:31.617 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.632+0000 7fd16affd640 1 -- 172.21.15.17:0/584204593 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd174036500 con 0x7fd174038c60 2024-03-29T11:23:31.617 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.633+0000 7fd16affd640 1 -- 172.21.15.17:0/584204593 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3946150189 0 0) 0x7fd174036730 con 0x7fd1740324b0 2024-03-29T11:23:31.617 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.633+0000 7fd16affd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:31.618 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.633+0000 7fd16affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:31.618 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.633+0000 7fd16affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:31.618 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.633+0000 7fd16affd640 10 monclient(hunting): my global_id is 4408 2024-03-29T11:23:31.618 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.633+0000 7fd16affd640 1 -- 172.21.15.17:0/584204593 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd17416e460 con 0x7fd1740324b0 2024-03-29T11:23:31.618 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.633+0000 7fd16affd640 1 -- 172.21.15.17:0/584204593 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 389677961 0 0) 0x7fd174036500 con 0x7fd174038c60 2024-03-29T11:23:31.618 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.633+0000 7fd16affd640 1 -- 172.21.15.17:0/584204593 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd174036730 con 0x7fd174038c60 2024-03-29T11:23:31.618 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.633+0000 7fd16affd640 1 -- 172.21.15.17:0/584204593 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fd170001670 con 0x7fd174038c60 2024-03-29T11:23:31.618 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.633+0000 7fd16affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:31.618 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.633+0000 7fd16affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:31.618 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.633+0000 7fd16affd640 10 monclient(hunting): dump: 2024-03-29T11:23:31.618 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:23:31.618 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:23:31.618 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:31.618 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:31.619 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:31.619 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:31.619 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:23:31.619 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:23:31.619 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:23:31.619 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:31.619 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.633+0000 7fd16affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:31.619 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.633+0000 7fd16affd640 1 -- 172.21.15.17:0/584204593 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3486299299 0 0) 0x7fd15c004eb0 con 0x7fd1740d11e0 2024-03-29T11:23:31.619 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.633+0000 7fd16affd640 1 -- 172.21.15.17:0/584204593 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd174036500 con 0x7fd1740d11e0 2024-03-29T11:23:31.619 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.633+0000 7fd16affd640 1 -- 172.21.15.17:0/584204593 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fd17c04fdd0 con 0x7fd1740d11e0 2024-03-29T11:23:31.619 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.633+0000 7fd16affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:31.619 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.633+0000 7fd16affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:31.619 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.633+0000 7fd16affd640 10 monclient(hunting): dump: 2024-03-29T11:23:31.619 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:23:31.619 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:23:31.619 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:31.619 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:31.620 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:31.620 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:31.620 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:23:31.620 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:23:31.620 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:23:31.620 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:31.620 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.633+0000 7fd16affd640 1 -- 172.21.15.17:0/584204593 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3525541401 0 0) 0x7fd17416e460 con 0x7fd1740324b0 2024-03-29T11:23:31.620 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.634+0000 7fd16affd640 1 -- 172.21.15.17:0/584204593 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd15c004eb0 con 0x7fd1740324b0 2024-03-29T11:23:31.620 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.634+0000 7fd16affd640 1 -- 172.21.15.17:0/584204593 <== mon.2 v1:172.21.15.17:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1837808494 0 0) 0x7fd174036730 con 0x7fd174038c60 2024-03-29T11:23:31.620 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.634+0000 7fd16affd640 1 -- 172.21.15.17:0/584204593 >> v1:172.21.15.94:6789/0 conn(0x7fd1740324b0 legacy=0x7fd1740c4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:31.620 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.634+0000 7fd16affd640 1 -- 172.21.15.17:0/584204593 >> v1:172.21.15.17:6789/0 conn(0x7fd1740d11e0 legacy=0x7fd1740348b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:31.620 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.634+0000 7fd16affd640 10 monclient: _finish_hunting 0 2024-03-29T11:23:31.620 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.634+0000 7fd16affd640 1 monclient: found mon.c 2024-03-29T11:23:31.620 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.634+0000 7fd16affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:31.620 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.634+0000 7fd16affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:23:31.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.634+0000 7fd16affd640 1 -- 172.21.15.17:0/584204593 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd17416f600 con 0x7fd174038c60 2024-03-29T11:23:31.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.634+0000 7fd16affd640 10 monclient: _finish_auth 0 2024-03-29T11:23:31.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.634+0000 7fd16affd640 10 monclient: _check_auth_tickets 2024-03-29T11:23:31.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.634+0000 7fd16affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:31.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.634+0000 7fd17b59e640 5 monclient: authenticate success, global_id 4421 2024-03-29T11:23:31.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.634+0000 7fd17b59e640 10 monclient: _renew_subs 2024-03-29T11:23:31.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.634+0000 7fd17b59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:23:31.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.634+0000 7fd17b59e640 1 -- 172.21.15.17:0/584204593 --> v1:172.21.15.17:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd17416e5f0 con 0x7fd174038c60 2024-03-29T11:23:31.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.634+0000 7fd17b59e640 10 client.4421.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:31.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.634+0000 7fd17b59e640 10 monclient: _renew_subs 2024-03-29T11:23:31.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.634+0000 7fd17b59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:23:31.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.634+0000 7fd17b59e640 1 -- 172.21.15.17:0/584204593 --> v1:172.21.15.17:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd17416ea80 con 0x7fd174038c60 2024-03-29T11:23:31.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.634+0000 7fd17b59e640 1 librados: init done 2024-03-29T11:23:31.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.634+0000 7fd16affd640 1 -- 172.21.15.17:0/584204593 <== mon.2 v1:172.21.15.17:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd170001150 con 0x7fd174038c60 2024-03-29T11:23:31.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.634+0000 7fd16affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:31.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.634+0000 7fd16affd640 1 -- 172.21.15.17:0/584204593 <== mon.2 v1:172.21.15.17:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fd170003bf0 con 0x7fd174038c60 2024-03-29T11:23:31.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.634+0000 7fd16affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:31.622 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.634+0000 7fd16affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:31.622 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.634+0000 7fd16affd640 10 monclient: dump: 2024-03-29T11:23:31.622 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:23:31.622 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:23:31.622 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:31.622 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:31.622 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:31.622 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:31.622 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:23:31.622 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:23:31.622 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:23:31.622 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:31.622 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.634+0000 7fd178d99640 4 set_mon_vals no callback set 2024-03-29T11:23:31.622 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.636+0000 7fd17b59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:31.622 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.636+0000 7fd17b59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:31.623 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.636+0000 7fd17b59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:23:31.623 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.636+0000 7fd17b59e640 1 -- 172.21.15.17:0/584204593 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd160005080 con 0x7fd174038c60 2024-03-29T11:23:31.623 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.638+0000 7fd16affd640 1 -- 172.21.15.17:0/584204593 <== mon.2 v1:172.21.15.17:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7fd170003d50 con 0x7fd174038c60 2024-03-29T11:23:31.624 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.639+0000 7fd17a59c640 10 monclient: get_auth_request con 0x7fd15c06dfc0 auth_method 0 2024-03-29T11:23:31.624 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.639+0000 7fd16affd640 1 -- 172.21.15.17:0/584204593 <== mon.2 v1:172.21.15.17:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7fd170084ab0 con 0x7fd174038c60 2024-03-29T11:23:31.624 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.639+0000 7fd16affd640 10 client.4421.objecter ms_dispatch 0x7fd17c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:31.624 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.639+0000 7fd16affd640 3 client.4421.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:31.624 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.639+0000 7fd16affd640 3 client.4421.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:31.624 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.639+0000 7fd16affd640 20 client.4421.objecter dump_active .. 0 homeless 2024-03-29T11:23:31.624 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.639+0000 7fd16affd640 10 client.4421.objecter ms_handle_connect 0x7fd15c06dfc0 2024-03-29T11:23:31.626 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.647+0000 7fd16affd640 1 -- 172.21.15.17:0/584204593 <== mon.2 v1:172.21.15.17:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd170050390 con 0x7fd174038c60 2024-03-29T11:23:31.626 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.647+0000 7fd16affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:31.626 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.647+0000 7fd16affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:31.961 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.982+0000 7fd17b59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:31.962 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.982+0000 7fd17b59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:31.962 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.982+0000 7fd17b59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:23:31.962 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.982+0000 7fd17b59e640 1 -- 172.21.15.17:0/584204593 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd160005330 con 0x7fd174038c60 2024-03-29T11:23:31.962 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.982+0000 7fd16affd640 1 -- 172.21.15.17:0/584204593 <== mon.2 v1:172.21.15.17:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7fd1700522c0 con 0x7fd174038c60 2024-03-29T11:23:31.962 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.982+0000 7fd16affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:31.962 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.982+0000 7fd16affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:31.962 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:23:31.962 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:31.966 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.985+0000 7fd17b59e640 10 librados: watch_flush enter 2024-03-29T11:23:31.967 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.987+0000 7fd17b59e640 10 librados: watch_flush exit 2024-03-29T11:23:31.967 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.987+0000 7fd17b59e640 20 client.4421.objecter shutdown clearing up homeless session... 2024-03-29T11:23:31.967 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.987+0000 7fd17b59e640 10 client.4421.objecter successfully canceled tick 2024-03-29T11:23:31.967 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.987+0000 7fd17b59e640 1 -- 172.21.15.17:0/584204593 >> v1:172.21.15.17:6816/3236185052 conn(0x7fd15c06dfc0 legacy=0x7fd15c070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:31.967 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.987+0000 7fd17b59e640 10 monclient: shutdown 2024-03-29T11:23:31.967 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.987+0000 7fd17b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:31.967 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.987+0000 7fd17b59e640 1 -- 172.21.15.17:0/584204593 >> v1:172.21.15.17:6790/0 conn(0x7fd174038c60 legacy=0x7fd174036c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:31.968 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.987+0000 7fd17b59e640 1 -- 172.21.15.17:0/584204593 shutdown_connections 2024-03-29T11:23:31.968 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.987+0000 7fd17b59e640 1 -- 172.21.15.17:0/584204593 >> 172.21.15.17:0/584204593 conn(0x7fd174005c20 msgr2=0x7fd1740043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:31.968 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.988+0000 7fd17b59e640 1 -- 172.21.15.17:0/584204593 shutdown_connections 2024-03-29T11:23:31.968 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.988+0000 7fd17b59e640 1 -- 172.21.15.17:0/584204593 wait complete. 2024-03-29T11:23:31.968 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:31.988+0000 7fd17b59e640 1 librados: shutdown 2024-03-29T11:23:31.983 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:38.789 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:38.926 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.946+0000 7f38cfa63640 10 monclient: get_monmap_and_config 2024-03-29T11:23:38.926 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.946+0000 7f38cfa63640 10 monclient: build_initial_monmap 2024-03-29T11:23:38.927 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.946+0000 7f38cfa63640 10 monclient: monmap: 2024-03-29T11:23:38.927 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:23:38.927 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:23:38.927 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:23:38.947953+0000 2024-03-29T11:23:38.927 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:23:38.947953+0000 2024-03-29T11:23:38.927 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:38.927 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:38.927 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:23:38.927 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:23:38.927 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:23:38.927 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:38.927 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.948+0000 7f38cfa63640 1 Processor -- start 2024-03-29T11:23:38.928 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.948+0000 7f38cfa63640 1 -- start start 2024-03-29T11:23:38.928 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.948+0000 7f38cfa63640 10 monclient: init 2024-03-29T11:23:38.929 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.949+0000 7f38cfa63640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:38.929 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.949+0000 7f38cfa63640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:23:38.929 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.949+0000 7f38cfa63640 10 monclient(hunting): picked mon.noname-c con 0x7f38c80b0860 addr v1:172.21.15.94:6789/0 2024-03-29T11:23:38.930 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.949+0000 7f38cfa63640 10 monclient(hunting): picked mon.noname-b con 0x7f38c8072450 addr v1:172.21.15.17:6790/0 2024-03-29T11:23:38.930 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.949+0000 7f38cfa63640 10 monclient(hunting): picked mon.noname-a con 0x7f38c813ee40 addr v1:172.21.15.17:6789/0 2024-03-29T11:23:38.930 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.949+0000 7f38cfa63640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38c80ad840 con 0x7f38c813ee40 2024-03-29T11:23:38.930 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.949+0000 7f38cfa63640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38c813c830 con 0x7f38c80b0860 2024-03-29T11:23:38.930 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.949+0000 7f38cfa63640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38c813c9c0 con 0x7f38c8072450 2024-03-29T11:23:38.930 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.949+0000 7f38cfa63640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:38.930 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.949+0000 7f38cfa63640 10 monclient(hunting): authenticate will time out at 1125.154472s 2024-03-29T11:23:38.930 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.949+0000 7f38ccffe640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f38c8072450 0x7f38c813c120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:60554/0 (socket says 172.21.15.17:60554) 2024-03-29T11:23:38.930 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.949+0000 7f38ccffe640 1 -- 172.21.15.17:0/3213742167 learned_addr learned my addr 172.21.15.17:0/3213742167 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:23:38.930 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.949+0000 7f38bbfff640 1 -- 172.21.15.17:0/3213742167 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 494886997 0 0) 0x7f38c813c9c0 con 0x7f38c8072450 2024-03-29T11:23:38.930 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.949+0000 7f38bbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:38.930 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.949+0000 7f38bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:38.930 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.949+0000 7f38bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:38.930 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.949+0000 7f38bbfff640 10 monclient(hunting): my global_id is 4424 2024-03-29T11:23:38.931 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.950+0000 7f38bbfff640 1 -- 172.21.15.17:0/3213742167 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38b4003520 con 0x7f38c8072450 2024-03-29T11:23:38.931 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.950+0000 7f38bbfff640 1 -- 172.21.15.17:0/3213742167 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1236747927 0 0) 0x7f38b4003520 con 0x7f38c8072450 2024-03-29T11:23:38.931 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.950+0000 7f38bbfff640 1 -- 172.21.15.17:0/3213742167 >> v1:172.21.15.94:6789/0 conn(0x7f38c80b0860 legacy=0x7f38c80acfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:38.931 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.950+0000 7f38bbfff640 1 -- 172.21.15.17:0/3213742167 >> v1:172.21.15.17:6789/0 conn(0x7f38c813ee40 legacy=0x7f38c813f260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:23:38.933 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.952+0000 7f38bbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:23:38.933 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.952+0000 7f38bbfff640 1 monclient: found mon.noname-b 2024-03-29T11:23:38.934 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.952+0000 7f38bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:38.934 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.952+0000 7f38bbfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.17:6790/0 2024-03-29T11:23:38.934 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.952+0000 7f38bbfff640 1 -- 172.21.15.17:0/3213742167 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c813f970 con 0x7f38c8072450 2024-03-29T11:23:38.934 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.952+0000 7f38bbfff640 10 monclient: _finish_auth 0 2024-03-29T11:23:38.934 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.952+0000 7f38bbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:23:38.934 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.952+0000 7f38bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:38.934 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.952+0000 7f38cfa63640 5 monclient: authenticate success, global_id 4424 2024-03-29T11:23:38.934 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.952+0000 7f38cfa63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:38.934 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.952+0000 7f38bbfff640 1 -- 172.21.15.17:0/3213742167 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f38bc001690 con 0x7f38c8072450 2024-03-29T11:23:38.934 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.952+0000 7f38bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:38.934 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.952+0000 7f38bbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:38.934 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.952+0000 7f38bbfff640 10 monclient: dump: 2024-03-29T11:23:38.934 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:23:38.934 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:23:38.935 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:38.935 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:38.935 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:38.935 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:38.935 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:23:38.935 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:23:38.935 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:23:38.935 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:38.935 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.952+0000 7f38cfa63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:38.935 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.953+0000 7f38bbfff640 1 -- 172.21.15.17:0/3213742167 <== mon.2 v1:172.21.15.17:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38bc001920 con 0x7f38c8072450 2024-03-29T11:23:38.935 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.953+0000 7f38bbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:38.935 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.953+0000 7f38cfa63640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:38.935 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.953+0000 7f38cfa63640 4 set_mon_vals no callback set 2024-03-29T11:23:38.935 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.954+0000 7f38bbfff640 1 -- 172.21.15.17:0/3213742167 <== mon.2 v1:172.21.15.17:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f38bc001cb0 con 0x7f38c8072450 2024-03-29T11:23:38.935 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.954+0000 7f38bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:38.936 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.954+0000 7f38bbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:38.936 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.954+0000 7f38bbfff640 10 monclient: dump: 2024-03-29T11:23:38.936 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:23:38.936 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:23:38.936 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:38.936 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:38.936 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:38.936 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:38.936 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:23:38.936 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:23:38.936 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:23:38.936 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:38.936 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.954+0000 7f38cfa63640 10 monclient: shutdown 2024-03-29T11:23:38.936 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.954+0000 7f38cfa63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:38.936 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.954+0000 7f38cfa63640 1 -- 172.21.15.17:0/3213742167 >> v1:172.21.15.17:6790/0 conn(0x7f38c8072450 legacy=0x7f38c813c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:38.937 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.954+0000 7f38cfa63640 1 -- 172.21.15.17:0/3213742167 shutdown_connections 2024-03-29T11:23:38.937 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.954+0000 7f38cfa63640 1 -- 172.21.15.17:0/3213742167 >> 172.21.15.17:0/3213742167 conn(0x7f38c806b660 msgr2=0x7f38c806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:38.939 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.954+0000 7f38cfa63640 1 -- 172.21.15.17:0/3213742167 shutdown_connections 2024-03-29T11:23:38.939 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.954+0000 7f38cfa63640 1 -- 172.21.15.17:0/3213742167 wait complete. 2024-03-29T11:23:38.939 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.956+0000 7f38cfa63640 10 monclient: build_initial_monmap 2024-03-29T11:23:38.940 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.956+0000 7f38cfa63640 10 monclient: monmap: 2024-03-29T11:23:38.940 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:23:38.940 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:38.940 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:23:38.940 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:23:38.940 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:38.940 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:38.940 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:23:38.940 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:23:38.940 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:23:38.940 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:38.940 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.956+0000 7f38cfa63640 1 librados: starting msgr at 2024-03-29T11:23:38.940 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.956+0000 7f38cfa63640 1 librados: starting objecter 2024-03-29T11:23:38.940 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.956+0000 7f38cfa63640 1 Processor -- start 2024-03-29T11:23:38.940 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.956+0000 7f38cfa63640 1 -- start start 2024-03-29T11:23:38.940 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.956+0000 7f38cfa63640 1 librados: setting wanted keys 2024-03-29T11:23:38.941 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.956+0000 7f38cfa63640 1 librados: calling monclient init 2024-03-29T11:23:38.941 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.956+0000 7f38cfa63640 10 monclient: init 2024-03-29T11:23:38.941 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.957+0000 7f38cfa63640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:38.941 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.957+0000 7f38cfa63640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:23:38.941 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.957+0000 7f38cfa63640 10 monclient(hunting): picked mon.noname-b con 0x7f38c8072450 addr v1:172.21.15.94:6789/0 2024-03-29T11:23:38.941 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.957+0000 7f38cfa63640 10 monclient(hunting): picked mon.noname-a con 0x7f38c80b0860 addr v1:172.21.15.17:6789/0 2024-03-29T11:23:38.941 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.957+0000 7f38cfa63640 10 monclient(hunting): picked mon.noname-c con 0x7f38c813ee40 addr v1:172.21.15.17:6790/0 2024-03-29T11:23:38.941 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.957+0000 7f38cfa63640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38c80b2100 con 0x7f38c80b0860 2024-03-29T11:23:38.941 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.957+0000 7f38cfa63640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38c80b1400 con 0x7f38c8072450 2024-03-29T11:23:38.941 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.957+0000 7f38cfa63640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38c80b2350 con 0x7f38c813ee40 2024-03-29T11:23:38.941 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.957+0000 7f38cfa63640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:38.941 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.957+0000 7f38cfa63640 10 monclient(hunting): authenticate will time out at 1125.162418s 2024-03-29T11:23:38.941 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.957+0000 7f38ce000640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f38c813ee40 0x7f38c81df700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:33102/0 (socket says 172.21.15.17:33102) 2024-03-29T11:23:38.941 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.957+0000 7f38ce000640 1 -- 172.21.15.17:0/1128240656 learned_addr learned my addr 172.21.15.17:0/1128240656 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:23:38.941 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.957+0000 7f38b9ffb640 10 client.?.objecter ms_handle_connect 0x7f38c813ee40 2024-03-29T11:23:38.941 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.957+0000 7f38b9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:38.942 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.957+0000 7f38b9ffb640 1 -- 172.21.15.17:0/1128240656 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3229235202 0 0) 0x7f38c80b2350 con 0x7f38c813ee40 2024-03-29T11:23:38.942 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.957+0000 7f38b9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:38.942 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.957+0000 7f38b9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:38.942 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.957+0000 7f38b9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:38.942 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.957+0000 7f38b9ffb640 10 monclient(hunting): my global_id is 4427 2024-03-29T11:23:38.942 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.957+0000 7f38b9ffb640 1 -- 172.21.15.17:0/1128240656 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38ac004eb0 con 0x7f38c813ee40 2024-03-29T11:23:38.942 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.958+0000 7f38b9ffb640 1 -- 172.21.15.17:0/1128240656 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4172084902 0 0) 0x7f38ac004eb0 con 0x7f38c813ee40 2024-03-29T11:23:38.942 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.958+0000 7f38b9ffb640 1 -- 172.21.15.17:0/1128240656 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38c80b2350 con 0x7f38c813ee40 2024-03-29T11:23:38.942 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.958+0000 7f38b9ffb640 1 -- 172.21.15.17:0/1128240656 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f38c4001750 con 0x7f38c813ee40 2024-03-29T11:23:38.942 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.958+0000 7f38b9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:38.942 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.958+0000 7f38b9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:38.942 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.958+0000 7f38b9ffb640 10 monclient(hunting): dump: 2024-03-29T11:23:38.942 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:23:38.942 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:23:38.942 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:38.942 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:38.943 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:38.943 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:38.943 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:23:38.943 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:23:38.943 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:23:38.943 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:38.943 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.958+0000 7f38b9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:38.943 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.958+0000 7f38b9ffb640 1 -- 172.21.15.17:0/1128240656 <== mon.2 v1:172.21.15.17:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3136148939 0 0) 0x7f38c80b2350 con 0x7f38c813ee40 2024-03-29T11:23:38.943 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.958+0000 7f38b9ffb640 1 -- 172.21.15.17:0/1128240656 >> v1:172.21.15.94:6789/0 conn(0x7f38c8072450 legacy=0x7f38c8129660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:38.943 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.958+0000 7f38b9ffb640 1 -- 172.21.15.17:0/1128240656 >> v1:172.21.15.17:6789/0 conn(0x7f38c80b0860 legacy=0x7f38c81deff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:23:38.943 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.960+0000 7f38b9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:23:38.943 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.960+0000 7f38b9ffb640 1 monclient: found mon.c 2024-03-29T11:23:38.943 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.960+0000 7f38b9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:38.943 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.960+0000 7f38b9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:23:38.943 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.960+0000 7f38b9ffb640 1 -- 172.21.15.17:0/1128240656 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c80b3b40 con 0x7f38c813ee40 2024-03-29T11:23:38.943 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.960+0000 7f38b9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:23:38.943 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.960+0000 7f38b9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:23:38.944 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.960+0000 7f38b9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:38.945 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.960+0000 7f38cfa63640 5 monclient: authenticate success, global_id 4427 2024-03-29T11:23:38.945 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.960+0000 7f38cfa63640 10 monclient: _renew_subs 2024-03-29T11:23:38.945 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.960+0000 7f38cfa63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:23:38.945 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.960+0000 7f38cfa63640 1 -- 172.21.15.17:0/1128240656 --> v1:172.21.15.17:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38c8137600 con 0x7f38c813ee40 2024-03-29T11:23:38.946 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.960+0000 7f38cfa63640 10 client.4427.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:38.946 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.960+0000 7f38cfa63640 10 monclient: _renew_subs 2024-03-29T11:23:38.946 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.960+0000 7f38cfa63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:23:38.946 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.960+0000 7f38cfa63640 1 -- 172.21.15.17:0/1128240656 --> v1:172.21.15.17:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f38c8129f50 con 0x7f38c813ee40 2024-03-29T11:23:38.946 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.960+0000 7f38cfa63640 1 librados: init done 2024-03-29T11:23:38.946 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.961+0000 7f38b9ffb640 1 -- 172.21.15.17:0/1128240656 <== mon.2 v1:172.21.15.17:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38c4001d20 con 0x7f38c813ee40 2024-03-29T11:23:38.946 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.961+0000 7f38b9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:38.946 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.961+0000 7f38b9ffb640 1 -- 172.21.15.17:0/1128240656 <== mon.2 v1:172.21.15.17:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f38c4004a00 con 0x7f38c813ee40 2024-03-29T11:23:38.946 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.961+0000 7f38b9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:38.946 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.961+0000 7f38b9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:38.946 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.961+0000 7f38b9ffb640 10 monclient: dump: 2024-03-29T11:23:38.946 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:23:38.946 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:23:38.946 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:38.946 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:38.947 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:38.947 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:38.947 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:23:38.947 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:23:38.947 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:23:38.947 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:38.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.961+0000 7f38bb7fe640 4 set_mon_vals no callback set 2024-03-29T11:23:38.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.962+0000 7f38b9ffb640 1 -- 172.21.15.17:0/1128240656 <== mon.2 v1:172.21.15.17:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f38c401aa50 con 0x7f38c813ee40 2024-03-29T11:23:38.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.962+0000 7f38cfa63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:38.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.963+0000 7f38b9ffb640 1 -- 172.21.15.17:0/1128240656 <== mon.2 v1:172.21.15.17:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f38c40849d0 con 0x7f38c813ee40 2024-03-29T11:23:38.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.963+0000 7f38b9ffb640 10 client.4427.objecter ms_dispatch 0x7f38c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:38.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.963+0000 7f38b9ffb640 3 client.4427.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:38.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.963+0000 7f38b9ffb640 3 client.4427.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:38.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.963+0000 7f38b9ffb640 20 client.4427.objecter dump_active .. 0 homeless 2024-03-29T11:23:38.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.963+0000 7f38cfa63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:38.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.963+0000 7f38cfa63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:23:38.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.963+0000 7f38cfa63640 1 -- 172.21.15.17:0/1128240656 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38a8005080 con 0x7f38c813ee40 2024-03-29T11:23:38.948 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.963+0000 7f38cd7ff640 10 monclient: get_auth_request con 0x7f38ac06dc40 auth_method 0 2024-03-29T11:23:38.948 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.963+0000 7f38b9ffb640 10 client.4427.objecter ms_handle_connect 0x7f38ac06dc40 2024-03-29T11:23:38.952 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.973+0000 7f38b9ffb640 1 -- 172.21.15.17:0/1128240656 <== mon.2 v1:172.21.15.17:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f38c4050310 con 0x7f38c813ee40 2024-03-29T11:23:38.952 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.973+0000 7f38b9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:38.952 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:38.973+0000 7f38b9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:39.257 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:39.278+0000 7f38cfa63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:39.258 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:39.278+0000 7f38cfa63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:39.258 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:39.278+0000 7f38cfa63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:23:39.258 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:39.278+0000 7f38cfa63640 1 -- 172.21.15.17:0/1128240656 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f38a8005330 con 0x7f38c813ee40 2024-03-29T11:23:39.258 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:39.278+0000 7f38b9ffb640 1 -- 172.21.15.17:0/1128240656 <== mon.2 v1:172.21.15.17:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f38c4052240 con 0x7f38c813ee40 2024-03-29T11:23:39.258 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:39.279+0000 7f38b9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:39.258 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:39.279+0000 7f38b9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:39.259 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:23:39.259 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:39.260 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:39.281+0000 7f38cfa63640 10 librados: watch_flush enter 2024-03-29T11:23:39.261 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:39.281+0000 7f38cfa63640 10 librados: watch_flush exit 2024-03-29T11:23:39.261 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:39.281+0000 7f38cfa63640 20 client.4427.objecter shutdown clearing up homeless session... 2024-03-29T11:23:39.261 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:39.281+0000 7f38cfa63640 10 client.4427.objecter successfully canceled tick 2024-03-29T11:23:39.261 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:39.282+0000 7f38cfa63640 1 -- 172.21.15.17:0/1128240656 >> v1:172.21.15.17:6816/3236185052 conn(0x7f38ac06dc40 legacy=0x7f38ac070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:39.261 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:39.282+0000 7f38cfa63640 10 monclient: shutdown 2024-03-29T11:23:39.261 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:39.282+0000 7f38cfa63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:39.261 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:39.282+0000 7f38cfa63640 1 -- 172.21.15.17:0/1128240656 >> v1:172.21.15.17:6790/0 conn(0x7f38c813ee40 legacy=0x7f38c81df700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:39.262 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:39.282+0000 7f38cfa63640 1 -- 172.21.15.17:0/1128240656 shutdown_connections 2024-03-29T11:23:39.262 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:39.282+0000 7f38cfa63640 1 -- 172.21.15.17:0/1128240656 >> 172.21.15.17:0/1128240656 conn(0x7f38c806b660 msgr2=0x7f38c806ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:39.262 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:39.283+0000 7f38cfa63640 1 -- 172.21.15.17:0/1128240656 shutdown_connections 2024-03-29T11:23:39.262 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:39.283+0000 7f38cfa63640 1 -- 172.21.15.17:0/1128240656 wait complete. 2024-03-29T11:23:39.262 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:39.283+0000 7f38cfa63640 1 librados: shutdown 2024-03-29T11:23:39.276 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:46.082 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:46.236 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.256+0000 7fdc30c4e640 10 monclient: get_monmap_and_config 2024-03-29T11:23:46.237 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.256+0000 7fdc30c4e640 10 monclient: build_initial_monmap 2024-03-29T11:23:46.237 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.257+0000 7fdc30c4e640 10 monclient: monmap: 2024-03-29T11:23:46.237 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:23:46.237 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:23:46.237 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:23:46.257975+0000 2024-03-29T11:23:46.237 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:23:46.257975+0000 2024-03-29T11:23:46.237 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:46.237 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:46.237 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:23:46.237 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:23:46.237 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:23:46.237 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:46.238 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.257+0000 7fdc30c4e640 1 Processor -- start 2024-03-29T11:23:46.241 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.258+0000 7fdc30c4e640 1 -- start start 2024-03-29T11:23:46.241 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.258+0000 7fdc30c4e640 10 monclient: init 2024-03-29T11:23:46.241 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.258+0000 7fdc30c4e640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:46.241 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.258+0000 7fdc30c4e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:23:46.241 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.258+0000 7fdc30c4e640 10 monclient(hunting): picked mon.noname-a con 0x7fdc2c1457f0 addr v1:172.21.15.17:6789/0 2024-03-29T11:23:46.241 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.258+0000 7fdc30c4e640 10 monclient(hunting): picked mon.noname-b con 0x7fdc2c13f650 addr v1:172.21.15.17:6790/0 2024-03-29T11:23:46.241 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.259+0000 7fdc30c4e640 10 monclient(hunting): picked mon.noname-c con 0x7fdc2c0a9d70 addr v1:172.21.15.94:6789/0 2024-03-29T11:23:46.241 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.259+0000 7fdc30c4e640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc2c0aaef0 con 0x7fdc2c1457f0 2024-03-29T11:23:46.241 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.259+0000 7fdc30c4e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc2c145bd0 con 0x7fdc2c0a9d70 2024-03-29T11:23:46.241 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.259+0000 7fdc30c4e640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc2c145eb0 con 0x7fdc2c13f650 2024-03-29T11:23:46.241 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.259+0000 7fdc30c4e640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:46.241 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.259+0000 7fdc30c4e640 10 monclient(hunting): authenticate will time out at 1132.463924s 2024-03-29T11:23:46.242 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.261+0000 7fdc2ad9d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fdc2c0a9d70 0x7fdc2c14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:44391/0 (socket says 172.21.15.17:44391) 2024-03-29T11:23:46.242 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.261+0000 7fdc2ad9d640 1 -- 172.21.15.17:0/1590676646 learned_addr learned my addr 172.21.15.17:0/1590676646 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:23:46.243 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.262+0000 7fdc2959a640 1 -- 172.21.15.17:0/1590676646 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 965427663 0 0) 0x7fdc2c145bd0 con 0x7fdc2c0a9d70 2024-03-29T11:23:46.243 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.262+0000 7fdc2959a640 10 monclient(hunting): _negotiate 2024-03-29T11:23:46.243 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.262+0000 7fdc2959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:46.243 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.262+0000 7fdc2959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:46.243 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.262+0000 7fdc2959a640 10 monclient(hunting): my global_id is 4414 2024-03-29T11:23:46.243 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.262+0000 7fdc2959a640 1 -- 172.21.15.17:0/1590676646 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc14003520 con 0x7fdc2c0a9d70 2024-03-29T11:23:46.244 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.264+0000 7fdc2959a640 1 -- 172.21.15.17:0/1590676646 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3956682110 0 0) 0x7fdc14003520 con 0x7fdc2c0a9d70 2024-03-29T11:23:46.244 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.264+0000 7fdc2959a640 1 -- 172.21.15.17:0/1590676646 >> v1:172.21.15.17:6790/0 conn(0x7fdc2c13f650 legacy=0x7fdc2c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:23:46.244 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.264+0000 7fdc2959a640 1 -- 172.21.15.17:0/1590676646 >> v1:172.21.15.17:6789/0 conn(0x7fdc2c1457f0 legacy=0x7fdc2c13ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:23:46.247 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.265+0000 7fdc2959a640 10 monclient: _finish_hunting 0 2024-03-29T11:23:46.247 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.265+0000 7fdc2959a640 1 monclient: found mon.noname-c 2024-03-29T11:23:46.247 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.265+0000 7fdc2959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:46.247 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.265+0000 7fdc2959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6789/0 2024-03-29T11:23:46.247 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.265+0000 7fdc2959a640 1 -- 172.21.15.17:0/1590676646 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc2c146340 con 0x7fdc2c0a9d70 2024-03-29T11:23:46.247 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.265+0000 7fdc2959a640 10 monclient: _finish_auth 0 2024-03-29T11:23:46.247 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.265+0000 7fdc2959a640 10 monclient: _check_auth_tickets 2024-03-29T11:23:46.247 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.265+0000 7fdc2959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:46.247 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.265+0000 7fdc30c4e640 5 monclient: authenticate success, global_id 4414 2024-03-29T11:23:46.247 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.265+0000 7fdc30c4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:46.247 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.265+0000 7fdc2959a640 1 -- 172.21.15.17:0/1590676646 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fdc24001a10 con 0x7fdc2c0a9d70 2024-03-29T11:23:46.247 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.265+0000 7fdc2959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:46.248 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.265+0000 7fdc2959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:46.248 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.265+0000 7fdc2959a640 10 monclient: dump: 2024-03-29T11:23:46.248 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:23:46.248 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:23:46.248 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:46.248 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:46.248 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:46.248 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:46.248 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:23:46.248 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:23:46.248 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:23:46.248 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:46.248 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.265+0000 7fdc30c4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:46.248 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.265+0000 7fdc2959a640 1 -- 172.21.15.17:0/1590676646 <== mon.1 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc24001ca0 con 0x7fdc2c0a9d70 2024-03-29T11:23:46.248 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.265+0000 7fdc2959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:46.248 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.265+0000 7fdc30c4e640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:46.249 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.265+0000 7fdc30c4e640 4 set_mon_vals no callback set 2024-03-29T11:23:46.249 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.265+0000 7fdc2959a640 1 -- 172.21.15.17:0/1590676646 <== mon.1 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fdc24002030 con 0x7fdc2c0a9d70 2024-03-29T11:23:46.249 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.265+0000 7fdc2959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:46.249 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.265+0000 7fdc2959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:46.249 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.265+0000 7fdc2959a640 10 monclient: dump: 2024-03-29T11:23:46.249 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:23:46.249 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:23:46.249 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:46.249 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:46.249 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:46.249 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:46.249 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:23:46.249 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:23:46.249 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:23:46.249 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:46.249 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.266+0000 7fdc30c4e640 10 monclient: shutdown 2024-03-29T11:23:46.250 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.266+0000 7fdc30c4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:46.250 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.266+0000 7fdc30c4e640 1 -- 172.21.15.17:0/1590676646 >> v1:172.21.15.94:6789/0 conn(0x7fdc2c0a9d70 legacy=0x7fdc2c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:46.250 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.268+0000 7fdc30c4e640 1 -- 172.21.15.17:0/1590676646 shutdown_connections 2024-03-29T11:23:46.250 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.268+0000 7fdc30c4e640 1 -- 172.21.15.17:0/1590676646 >> 172.21.15.17:0/1590676646 conn(0x7fdc2c066a00 msgr2=0x7fdc2c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:46.250 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.268+0000 7fdc30c4e640 1 -- 172.21.15.17:0/1590676646 shutdown_connections 2024-03-29T11:23:46.251 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.268+0000 7fdc30c4e640 1 -- 172.21.15.17:0/1590676646 wait complete. 2024-03-29T11:23:46.254 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.269+0000 7fdc30c4e640 10 monclient: build_initial_monmap 2024-03-29T11:23:46.254 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.269+0000 7fdc30c4e640 10 monclient: monmap: 2024-03-29T11:23:46.254 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:23:46.254 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:46.254 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:23:46.254 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:23:46.254 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:46.254 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:46.254 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:23:46.254 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:23:46.254 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:23:46.255 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:46.255 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.270+0000 7fdc30c4e640 1 librados: starting msgr at 2024-03-29T11:23:46.255 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.270+0000 7fdc30c4e640 1 librados: starting objecter 2024-03-29T11:23:46.255 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.270+0000 7fdc30c4e640 1 Processor -- start 2024-03-29T11:23:46.255 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.274+0000 7fdc30c4e640 1 -- start start 2024-03-29T11:23:46.255 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.274+0000 7fdc30c4e640 1 librados: setting wanted keys 2024-03-29T11:23:46.255 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.274+0000 7fdc30c4e640 1 librados: calling monclient init 2024-03-29T11:23:46.255 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.274+0000 7fdc30c4e640 10 monclient: init 2024-03-29T11:23:46.256 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.275+0000 7fdc30c4e640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:46.256 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.275+0000 7fdc30c4e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:23:46.256 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.275+0000 7fdc30c4e640 10 monclient(hunting): picked mon.noname-b con 0x7fdc2c0a9d70 addr v1:172.21.15.94:6789/0 2024-03-29T11:23:46.256 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.275+0000 7fdc30c4e640 10 monclient(hunting): picked mon.noname-a con 0x7fdc2c13f650 addr v1:172.21.15.17:6789/0 2024-03-29T11:23:46.256 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.275+0000 7fdc30c4e640 10 monclient(hunting): picked mon.noname-c con 0x7fdc2c1457f0 addr v1:172.21.15.17:6790/0 2024-03-29T11:23:46.256 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.275+0000 7fdc30c4e640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc2c1d8220 con 0x7fdc2c13f650 2024-03-29T11:23:46.257 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.275+0000 7fdc30c4e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc2c06f040 con 0x7fdc2c0a9d70 2024-03-29T11:23:46.257 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.275+0000 7fdc30c4e640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc2c06f320 con 0x7fdc2c1457f0 2024-03-29T11:23:46.257 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.275+0000 7fdc30c4e640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:46.257 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.275+0000 7fdc30c4e640 10 monclient(hunting): authenticate will time out at 1132.480223s 2024-03-29T11:23:46.265 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.280+0000 7fdc2a59c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fdc2c0a9d70 0x7fdc2c13fda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:44483/0 (socket says 172.21.15.17:44483) 2024-03-29T11:23:46.265 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.280+0000 7fdc2a59c640 1 -- 172.21.15.17:0/1736233900 learned_addr learned my addr 172.21.15.17:0/1736233900 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:23:46.265 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.282+0000 7fdc1affd640 10 client.?.objecter ms_handle_connect 0x7fdc2c0a9d70 2024-03-29T11:23:46.265 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.282+0000 7fdc1affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:46.265 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.282+0000 7fdc1affd640 1 -- 172.21.15.17:0/1736233900 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1775669335 0 0) 0x7fdc2c06f040 con 0x7fdc2c0a9d70 2024-03-29T11:23:46.266 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.282+0000 7fdc1affd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:46.266 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.282+0000 7fdc1affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:46.266 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.282+0000 7fdc1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:46.266 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.282+0000 7fdc1affd640 10 monclient(hunting): my global_id is 4417 2024-03-29T11:23:46.266 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.282+0000 7fdc1affd640 1 -- 172.21.15.17:0/1736233900 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc0c004eb0 con 0x7fdc2c0a9d70 2024-03-29T11:23:46.266 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.282+0000 7fdc1affd640 1 -- 172.21.15.17:0/1736233900 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2864180732 0 0) 0x7fdc0c004eb0 con 0x7fdc2c0a9d70 2024-03-29T11:23:46.266 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.282+0000 7fdc1affd640 1 -- 172.21.15.17:0/1736233900 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdc2c06f040 con 0x7fdc2c0a9d70 2024-03-29T11:23:46.266 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.282+0000 7fdc1affd640 1 -- 172.21.15.17:0/1736233900 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fdc1c0017d0 con 0x7fdc2c0a9d70 2024-03-29T11:23:46.266 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.282+0000 7fdc1affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:46.266 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.282+0000 7fdc1affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:46.266 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.282+0000 7fdc1affd640 10 monclient(hunting): dump: 2024-03-29T11:23:46.266 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:23:46.266 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:23:46.266 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:46.267 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:46.267 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:46.267 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:46.267 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:23:46.267 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:23:46.267 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:23:46.267 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:46.267 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.282+0000 7fdc1affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:46.267 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.283+0000 7fdc1affd640 1 -- 172.21.15.17:0/1736233900 <== mon.1 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1333319642 0 0) 0x7fdc2c06f040 con 0x7fdc2c0a9d70 2024-03-29T11:23:46.267 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.283+0000 7fdc1affd640 1 -- 172.21.15.17:0/1736233900 >> v1:172.21.15.17:6790/0 conn(0x7fdc2c1457f0 legacy=0x7fdc2c1d7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:46.267 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.283+0000 7fdc1affd640 1 -- 172.21.15.17:0/1736233900 >> v1:172.21.15.17:6789/0 conn(0x7fdc2c13f650 legacy=0x7fdc2c1d7400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:46.267 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.283+0000 7fdc1affd640 10 monclient: _finish_hunting 0 2024-03-29T11:23:46.267 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.283+0000 7fdc1affd640 1 monclient: found mon.b 2024-03-29T11:23:46.267 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.284+0000 7fdc1affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:46.267 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.284+0000 7fdc1affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:23:46.267 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.284+0000 7fdc1affd640 1 -- 172.21.15.17:0/1736233900 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc2c142490 con 0x7fdc2c0a9d70 2024-03-29T11:23:46.267 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.284+0000 7fdc1affd640 10 monclient: _finish_auth 0 2024-03-29T11:23:46.268 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.284+0000 7fdc1affd640 10 monclient: _check_auth_tickets 2024-03-29T11:23:46.268 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.284+0000 7fdc1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:46.268 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.284+0000 7fdc30c4e640 5 monclient: authenticate success, global_id 4417 2024-03-29T11:23:46.268 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.284+0000 7fdc30c4e640 10 monclient: _renew_subs 2024-03-29T11:23:46.268 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.284+0000 7fdc30c4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:23:46.268 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.284+0000 7fdc30c4e640 1 -- 172.21.15.17:0/1736233900 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc2c142990 con 0x7fdc2c0a9d70 2024-03-29T11:23:46.268 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.284+0000 7fdc30c4e640 10 client.4417.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:46.268 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.284+0000 7fdc30c4e640 10 monclient: _renew_subs 2024-03-29T11:23:46.268 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.284+0000 7fdc30c4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:23:46.268 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.284+0000 7fdc30c4e640 1 -- 172.21.15.17:0/1736233900 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc2c141f90 con 0x7fdc2c0a9d70 2024-03-29T11:23:46.268 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.284+0000 7fdc30c4e640 1 librados: init done 2024-03-29T11:23:46.269 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.288+0000 7fdc1affd640 1 -- 172.21.15.17:0/1736233900 <== mon.1 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc1c001930 con 0x7fdc2c0a9d70 2024-03-29T11:23:46.269 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.288+0000 7fdc1affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:46.269 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.289+0000 7fdc28d99640 4 set_mon_vals no callback set 2024-03-29T11:23:46.270 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.289+0000 7fdc1affd640 1 -- 172.21.15.17:0/1736233900 <== mon.1 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fdc1c004a40 con 0x7fdc2c0a9d70 2024-03-29T11:23:46.270 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.290+0000 7fdc1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:46.270 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.290+0000 7fdc1affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:46.270 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.290+0000 7fdc1affd640 10 monclient: dump: 2024-03-29T11:23:46.270 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:23:46.270 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:23:46.270 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:46.270 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:46.270 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:46.270 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:46.270 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:23:46.270 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:23:46.270 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:23:46.271 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:46.271 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.290+0000 7fdc30c4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:46.272 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.291+0000 7fdc30c4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:46.272 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.291+0000 7fdc30c4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:23:46.272 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.291+0000 7fdc30c4e640 1 -- 172.21.15.17:0/1736233900 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc10005080 con 0x7fdc2c0a9d70 2024-03-29T11:23:46.272 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.291+0000 7fdc1affd640 1 -- 172.21.15.17:0/1736233900 <== mon.1 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7fdc1c01aad0 con 0x7fdc2c0a9d70 2024-03-29T11:23:46.272 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.292+0000 7fdc1affd640 1 -- 172.21.15.17:0/1736233900 <== mon.1 v1:172.21.15.94:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7fdc1c084ac0 con 0x7fdc2c0a9d70 2024-03-29T11:23:46.273 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.292+0000 7fdc1affd640 10 client.4417.objecter ms_dispatch 0x7fdc2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:46.273 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.292+0000 7fdc1affd640 3 client.4417.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:46.273 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.292+0000 7fdc1affd640 3 client.4417.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:46.273 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.292+0000 7fdc29d9b640 10 monclient: get_auth_request con 0x7fdc0c06dcc0 auth_method 0 2024-03-29T11:23:46.273 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.292+0000 7fdc1affd640 20 client.4417.objecter dump_active .. 0 homeless 2024-03-29T11:23:46.273 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.292+0000 7fdc1affd640 10 client.4417.objecter ms_handle_connect 0x7fdc0c06dcc0 2024-03-29T11:23:46.279 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.299+0000 7fdc1affd640 1 -- 172.21.15.17:0/1736233900 <== mon.1 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdc1c0501d0 con 0x7fdc2c0a9d70 2024-03-29T11:23:46.279 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.299+0000 7fdc1affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:46.280 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.299+0000 7fdc1affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:46.612 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.632+0000 7fdc30c4e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:46.612 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.632+0000 7fdc30c4e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:46.612 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.632+0000 7fdc30c4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:23:46.612 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.632+0000 7fdc30c4e640 1 -- 172.21.15.17:0/1736233900 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdc10005330 con 0x7fdc2c0a9d70 2024-03-29T11:23:46.614 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.634+0000 7fdc1affd640 1 -- 172.21.15.17:0/1736233900 <== mon.1 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7fdc1c0522d0 con 0x7fdc2c0a9d70 2024-03-29T11:23:46.614 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.634+0000 7fdc1affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:46.614 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.634+0000 7fdc1affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:46.614 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:23:46.614 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:46.617 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.637+0000 7fdc30c4e640 10 librados: watch_flush enter 2024-03-29T11:23:46.618 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.638+0000 7fdc30c4e640 10 librados: watch_flush exit 2024-03-29T11:23:46.618 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.638+0000 7fdc30c4e640 20 client.4417.objecter shutdown clearing up homeless session... 2024-03-29T11:23:46.618 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.638+0000 7fdc30c4e640 10 client.4417.objecter successfully canceled tick 2024-03-29T11:23:46.618 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.638+0000 7fdc30c4e640 1 -- 172.21.15.17:0/1736233900 >> v1:172.21.15.17:6816/3236185052 conn(0x7fdc0c06dcc0 legacy=0x7fdc0c070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:46.618 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.638+0000 7fdc30c4e640 10 monclient: shutdown 2024-03-29T11:23:46.618 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.638+0000 7fdc30c4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:46.619 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.638+0000 7fdc30c4e640 1 -- 172.21.15.17:0/1736233900 >> v1:172.21.15.94:6789/0 conn(0x7fdc2c0a9d70 legacy=0x7fdc2c13fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:46.619 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.640+0000 7fdc30c4e640 1 -- 172.21.15.17:0/1736233900 shutdown_connections 2024-03-29T11:23:46.620 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.640+0000 7fdc30c4e640 1 -- 172.21.15.17:0/1736233900 >> 172.21.15.17:0/1736233900 conn(0x7fdc2c066a00 msgr2=0x7fdc2c0b61c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:46.620 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.640+0000 7fdc30c4e640 1 -- 172.21.15.17:0/1736233900 shutdown_connections 2024-03-29T11:23:46.620 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.640+0000 7fdc30c4e640 1 -- 172.21.15.17:0/1736233900 wait complete. 2024-03-29T11:23:46.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:46.641+0000 7fdc30c4e640 1 librados: shutdown 2024-03-29T11:23:46.639 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:53.446 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:53.596 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.615+0000 7f1772cff640 10 monclient: get_monmap_and_config 2024-03-29T11:23:53.597 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.615+0000 7f1772cff640 10 monclient: build_initial_monmap 2024-03-29T11:23:53.597 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.615+0000 7f1772cff640 10 monclient: monmap: 2024-03-29T11:23:53.597 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:23:53.597 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:23:53.597 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:23:53.616145+0000 2024-03-29T11:23:53.597 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:23:53.616145+0000 2024-03-29T11:23:53.597 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:53.597 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:53.597 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:23:53.597 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:23:53.597 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:23:53.597 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:53.597 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.616+0000 7f1772cff640 1 Processor -- start 2024-03-29T11:23:53.598 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.616+0000 7f1772cff640 1 -- start start 2024-03-29T11:23:53.598 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.616+0000 7f1772cff640 10 monclient: init 2024-03-29T11:23:53.599 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.617+0000 7f1772cff640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:53.599 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.617+0000 7f1772cff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:23:53.599 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.617+0000 7f1772cff640 10 monclient(hunting): picked mon.noname-c con 0x7f176c0ad6c0 addr v1:172.21.15.94:6789/0 2024-03-29T11:23:53.599 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.617+0000 7f1772cff640 10 monclient(hunting): picked mon.noname-a con 0x7f176c152980 addr v1:172.21.15.17:6789/0 2024-03-29T11:23:53.599 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.617+0000 7f1772cff640 10 monclient(hunting): picked mon.noname-b con 0x7f176c1588d0 addr v1:172.21.15.17:6790/0 2024-03-29T11:23:53.599 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.617+0000 7f1772cff640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f176c0a94e0 con 0x7f176c152980 2024-03-29T11:23:53.599 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.617+0000 7f1772cff640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f176c0af3a0 con 0x7f176c0ad6c0 2024-03-29T11:23:53.599 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.617+0000 7f1772cff640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f176c0afb60 con 0x7f176c1588d0 2024-03-29T11:23:53.599 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.617+0000 7f1772cff640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:53.600 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.617+0000 7f1772cff640 10 monclient(hunting): authenticate will time out at 1139.822377s 2024-03-29T11:23:53.602 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.622+0000 7f1770a9b640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f176c0ad6c0 0x7f176c14fb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:47755/0 (socket says 172.21.15.17:47755) 2024-03-29T11:23:53.602 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.622+0000 7f1770a9b640 1 -- 172.21.15.17:0/3593690646 learned_addr learned my addr 172.21.15.17:0/3593690646 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:23:53.604 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.623+0000 7f176b7fe640 1 -- 172.21.15.17:0/3593690646 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1047974825 0 0) 0x7f176c0af3a0 con 0x7f176c0ad6c0 2024-03-29T11:23:53.604 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.623+0000 7f176b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:23:53.604 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.623+0000 7f176b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:53.604 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.623+0000 7f176b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:53.604 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.623+0000 7f176b7fe640 10 monclient(hunting): my global_id is 4420 2024-03-29T11:23:53.604 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.623+0000 7f176b7fe640 1 -- 172.21.15.17:0/3593690646 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1754003520 con 0x7f176c0ad6c0 2024-03-29T11:23:53.604 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.623+0000 7f176b7fe640 1 -- 172.21.15.17:0/3593690646 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3519099251 0 0) 0x7f1754003520 con 0x7f176c0ad6c0 2024-03-29T11:23:53.604 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.623+0000 7f176b7fe640 1 -- 172.21.15.17:0/3593690646 >> v1:172.21.15.17:6790/0 conn(0x7f176c1588d0 legacy=0x7f176c15acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:53.604 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.624+0000 7f176b7fe640 1 -- 172.21.15.17:0/3593690646 >> v1:172.21.15.17:6789/0 conn(0x7f176c152980 legacy=0x7f176c152df0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:23:53.607 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.625+0000 7f176b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:23:53.607 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.625+0000 7f176b7fe640 1 monclient: found mon.noname-c 2024-03-29T11:23:53.607 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.625+0000 7f176b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:53.607 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.625+0000 7f176b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6789/0 2024-03-29T11:23:53.607 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.625+0000 7f176b7fe640 1 -- 172.21.15.17:0/3593690646 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f176c0aab80 con 0x7f176c0ad6c0 2024-03-29T11:23:53.607 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.625+0000 7f176b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:23:53.607 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.626+0000 7f176b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:23:53.607 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.626+0000 7f176b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:53.607 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.626+0000 7f1772cff640 5 monclient: authenticate success, global_id 4420 2024-03-29T11:23:53.607 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.626+0000 7f1772cff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:53.607 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.626+0000 7f176b7fe640 1 -- 172.21.15.17:0/3593690646 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f1764001a10 con 0x7f176c0ad6c0 2024-03-29T11:23:53.608 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.626+0000 7f176b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:53.608 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.626+0000 7f176b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:53.608 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.626+0000 7f176b7fe640 10 monclient: dump: 2024-03-29T11:23:53.608 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:23:53.608 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:23:53.608 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:53.608 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:53.608 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:53.608 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:53.608 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:23:53.608 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:23:53.608 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:23:53.608 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:53.608 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.626+0000 7f1772cff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:53.609 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.626+0000 7f176b7fe640 1 -- 172.21.15.17:0/3593690646 <== mon.1 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1764001ca0 con 0x7f176c0ad6c0 2024-03-29T11:23:53.609 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.626+0000 7f176b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:53.610 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.626+0000 7f1772cff640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:53.610 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.626+0000 7f1772cff640 4 set_mon_vals no callback set 2024-03-29T11:23:53.610 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.627+0000 7f1772cff640 10 monclient: shutdown 2024-03-29T11:23:53.610 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.627+0000 7f1772cff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:53.611 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.627+0000 7f1772cff640 1 -- 172.21.15.17:0/3593690646 >> v1:172.21.15.94:6789/0 conn(0x7f176c0ad6c0 legacy=0x7f176c14fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:53.611 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.628+0000 7f1772cff640 1 -- 172.21.15.17:0/3593690646 shutdown_connections 2024-03-29T11:23:53.611 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.628+0000 7f1772cff640 1 -- 172.21.15.17:0/3593690646 >> 172.21.15.17:0/3593690646 conn(0x7f176c0b35e0 msgr2=0x7f176c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:53.611 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.628+0000 7f1772cff640 1 -- 172.21.15.17:0/3593690646 shutdown_connections 2024-03-29T11:23:53.611 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.628+0000 7f1772cff640 1 -- 172.21.15.17:0/3593690646 wait complete. 2024-03-29T11:23:53.611 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.629+0000 7f1772cff640 10 monclient: build_initial_monmap 2024-03-29T11:23:53.611 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.629+0000 7f1772cff640 10 monclient: monmap: 2024-03-29T11:23:53.611 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:23:53.611 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:53.611 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:23:53.611 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:23:53.611 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:53.611 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:53.611 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:23:53.612 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:23:53.612 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:23:53.612 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:53.612 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.629+0000 7f1772cff640 1 librados: starting msgr at 2024-03-29T11:23:53.612 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.629+0000 7f1772cff640 1 librados: starting objecter 2024-03-29T11:23:53.612 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.629+0000 7f1772cff640 1 Processor -- start 2024-03-29T11:23:53.612 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.629+0000 7f1772cff640 1 -- start start 2024-03-29T11:23:53.612 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.629+0000 7f1772cff640 1 librados: setting wanted keys 2024-03-29T11:23:53.612 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.629+0000 7f1772cff640 1 librados: calling monclient init 2024-03-29T11:23:53.612 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.629+0000 7f1772cff640 10 monclient: init 2024-03-29T11:23:53.613 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.630+0000 7f1772cff640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:53.613 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.630+0000 7f1772cff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:23:53.613 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.630+0000 7f1772cff640 10 monclient(hunting): picked mon.noname-c con 0x7f176c0ad6c0 addr v1:172.21.15.17:6790/0 2024-03-29T11:23:53.613 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.630+0000 7f1772cff640 10 monclient(hunting): picked mon.noname-a con 0x7f176c152980 addr v1:172.21.15.17:6789/0 2024-03-29T11:23:53.613 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.630+0000 7f1772cff640 10 monclient(hunting): picked mon.noname-b con 0x7f176c1588d0 addr v1:172.21.15.94:6789/0 2024-03-29T11:23:53.613 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.630+0000 7f1772cff640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f176c08ab80 con 0x7f176c152980 2024-03-29T11:23:53.613 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.630+0000 7f1772cff640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f176c086d60 con 0x7f176c1588d0 2024-03-29T11:23:53.613 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.630+0000 7f1772cff640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f176c1ecdf0 con 0x7f176c0ad6c0 2024-03-29T11:23:53.613 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.630+0000 7f1772cff640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:53.613 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.631+0000 7f1772cff640 10 monclient(hunting): authenticate will time out at 1139.835884s 2024-03-29T11:23:53.614 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.634+0000 7f176bfff640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f176c152980 0x7f176c1ec6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:45879/0 (socket says 172.21.15.17:45879) 2024-03-29T11:23:53.614 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.634+0000 7f176bfff640 1 -- 172.21.15.17:0/848824093 learned_addr learned my addr 172.21.15.17:0/848824093 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:23:53.615 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.635+0000 7f17697fa640 10 client.?.objecter ms_handle_connect 0x7f176c152980 2024-03-29T11:23:53.615 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.635+0000 7f17697fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:53.615 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.635+0000 7f17697fa640 1 -- 172.21.15.17:0/848824093 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3709814862 0 0) 0x7f176c08ab80 con 0x7f176c152980 2024-03-29T11:23:53.616 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.635+0000 7f17697fa640 10 monclient(hunting): _negotiate 2024-03-29T11:23:53.616 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.635+0000 7f17697fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:53.616 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.635+0000 7f17697fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:53.616 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.635+0000 7f17697fa640 10 monclient(hunting): my global_id is 4401 2024-03-29T11:23:53.616 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.636+0000 7f17697fa640 1 -- 172.21.15.17:0/848824093 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1758004eb0 con 0x7f176c152980 2024-03-29T11:23:53.616 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.636+0000 7f17697fa640 1 -- 172.21.15.17:0/848824093 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4155903974 0 0) 0x7f1758004eb0 con 0x7f176c152980 2024-03-29T11:23:53.616 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.636+0000 7f17697fa640 1 -- 172.21.15.17:0/848824093 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f176c08ab80 con 0x7f176c152980 2024-03-29T11:23:53.616 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.636+0000 7f17697fa640 1 -- 172.21.15.17:0/848824093 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f175c004080 con 0x7f176c152980 2024-03-29T11:23:53.616 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.636+0000 7f17697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:53.616 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.636+0000 7f17697fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:53.617 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.636+0000 7f17697fa640 10 monclient(hunting): dump: 2024-03-29T11:23:53.617 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:23:53.617 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:23:53.617 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:53.617 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:53.617 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:53.617 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:53.617 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:23:53.617 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:23:53.617 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:23:53.618 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:53.618 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.636+0000 7f17697fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:53.618 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.636+0000 7f17697fa640 1 -- 172.21.15.17:0/848824093 <== mon.0 v1:172.21.15.17:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3724098327 0 0) 0x7f176c08ab80 con 0x7f176c152980 2024-03-29T11:23:53.618 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.636+0000 7f17697fa640 1 -- 172.21.15.17:0/848824093 >> v1:172.21.15.17:6790/0 conn(0x7f176c0ad6c0 legacy=0x7f176c08a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:53.618 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.636+0000 7f17697fa640 1 -- 172.21.15.17:0/848824093 >> v1:172.21.15.94:6789/0 conn(0x7f176c1588d0 legacy=0x7f176c1e9a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:53.618 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.636+0000 7f17697fa640 10 monclient: _finish_hunting 0 2024-03-29T11:23:53.618 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.636+0000 7f17697fa640 1 monclient: found mon.a 2024-03-29T11:23:53.618 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.637+0000 7f17697fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:53.619 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.637+0000 7f17697fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:23:53.619 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.637+0000 7f17697fa640 1 -- 172.21.15.17:0/848824093 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f176c1ea190 con 0x7f176c152980 2024-03-29T11:23:53.619 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.637+0000 7f17697fa640 10 monclient: _finish_auth 0 2024-03-29T11:23:53.619 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.637+0000 7f17697fa640 10 monclient: _check_auth_tickets 2024-03-29T11:23:53.619 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.637+0000 7f17697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:53.619 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.637+0000 7f17697fa640 1 -- 172.21.15.17:0/848824093 <== mon.0 v1:172.21.15.17:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f175c0043a0 con 0x7f176c152980 2024-03-29T11:23:53.619 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.637+0000 7f17697fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:53.619 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.637+0000 7f17697fa640 1 -- 172.21.15.17:0/848824093 <== mon.0 v1:172.21.15.17:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f175c001320 con 0x7f176c152980 2024-03-29T11:23:53.619 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.637+0000 7f17697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:53.619 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.637+0000 7f17697fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:53.619 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.637+0000 7f17697fa640 10 monclient: dump: 2024-03-29T11:23:53.619 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:23:53.619 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:23:53.619 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:53.619 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:23:53.620 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:53.620 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:23:53.620 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:23:53.620 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:23:53.620 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:23:53.620 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:23:53.620 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.639+0000 7f176a7fc640 4 set_mon_vals no callback set 2024-03-29T11:23:53.620 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.639+0000 7f1772cff640 5 monclient: authenticate success, global_id 4401 2024-03-29T11:23:53.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.639+0000 7f1772cff640 10 monclient: _renew_subs 2024-03-29T11:23:53.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.639+0000 7f1772cff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:23:53.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.639+0000 7f1772cff640 1 -- 172.21.15.17:0/848824093 --> v1:172.21.15.17:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f176c086840 con 0x7f176c152980 2024-03-29T11:23:53.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.639+0000 7f1772cff640 10 client.4401.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:53.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.639+0000 7f1772cff640 10 monclient: _renew_subs 2024-03-29T11:23:53.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.639+0000 7f1772cff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:23:53.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.639+0000 7f1772cff640 1 -- 172.21.15.17:0/848824093 --> v1:172.21.15.17:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f176c087460 con 0x7f176c152980 2024-03-29T11:23:53.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.639+0000 7f1772cff640 1 librados: init done 2024-03-29T11:23:53.622 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.641+0000 7f17697fa640 1 -- 172.21.15.17:0/848824093 <== mon.0 v1:172.21.15.17:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f175c001480 con 0x7f176c152980 2024-03-29T11:23:53.622 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.641+0000 7f1772cff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:53.622 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.641+0000 7f1772cff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:53.622 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.641+0000 7f1772cff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:23:53.622 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.641+0000 7f1772cff640 1 -- 172.21.15.17:0/848824093 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f176c086f10 con 0x7f176c152980 2024-03-29T11:23:53.623 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.643+0000 7f1770a9b640 10 monclient: get_auth_request con 0x7f175806dc00 auth_method 0 2024-03-29T11:23:53.624 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.643+0000 7f17697fa640 1 -- 172.21.15.17:0/848824093 <== mon.0 v1:172.21.15.17:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f175c084b40 con 0x7f176c152980 2024-03-29T11:23:53.624 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.643+0000 7f17697fa640 10 client.4401.objecter ms_dispatch 0x7f176c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:53.624 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.643+0000 7f17697fa640 3 client.4401.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:53.624 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.643+0000 7f17697fa640 3 client.4401.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:53.624 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.643+0000 7f17697fa640 20 client.4401.objecter dump_active .. 0 homeless 2024-03-29T11:23:53.624 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.643+0000 7f17697fa640 10 client.4401.objecter ms_handle_connect 0x7f175806dc00 2024-03-29T11:23:53.635 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.653+0000 7f17697fa640 1 -- 172.21.15.17:0/848824093 <== mon.0 v1:172.21.15.17:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f176c086f10 con 0x7f176c152980 2024-03-29T11:23:53.635 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.653+0000 7f17697fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:53.635 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.653+0000 7f17697fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:53.954 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.974+0000 7f1772cff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:53.954 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.974+0000 7f1772cff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:53.955 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.974+0000 7f1772cff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:23:53.955 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.974+0000 7f1772cff640 1 -- 172.21.15.17:0/848824093 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f176c086f10 con 0x7f176c152980 2024-03-29T11:23:53.955 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.975+0000 7f17697fa640 1 -- 172.21.15.17:0/848824093 <== mon.0 v1:172.21.15.17:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f176c086f10 con 0x7f176c152980 2024-03-29T11:23:53.955 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.975+0000 7f17697fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:53.955 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.975+0000 7f17697fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:53.959 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:23:53.959 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:53.964 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.982+0000 7f1772cff640 10 librados: watch_flush enter 2024-03-29T11:23:53.964 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.982+0000 7f1772cff640 10 librados: watch_flush exit 2024-03-29T11:23:53.964 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.982+0000 7f1772cff640 20 client.4401.objecter shutdown clearing up homeless session... 2024-03-29T11:23:53.964 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.982+0000 7f1772cff640 10 client.4401.objecter successfully canceled tick 2024-03-29T11:23:53.964 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.982+0000 7f1772cff640 1 -- 172.21.15.17:0/848824093 >> v1:172.21.15.17:6816/3236185052 conn(0x7f175806dc00 legacy=0x7f17580700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:53.964 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.982+0000 7f1772cff640 10 monclient: shutdown 2024-03-29T11:23:53.965 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.982+0000 7f1772cff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:53.965 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.982+0000 7f1772cff640 1 -- 172.21.15.17:0/848824093 >> v1:172.21.15.17:6789/0 conn(0x7f176c152980 legacy=0x7f176c1ec6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:53.965 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.984+0000 7f1772cff640 1 -- 172.21.15.17:0/848824093 shutdown_connections 2024-03-29T11:23:53.965 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.984+0000 7f1772cff640 1 -- 172.21.15.17:0/848824093 >> 172.21.15.17:0/848824093 conn(0x7f176c0b35e0 msgr2=0x7f176c0ab380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:53.966 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.985+0000 7f1772cff640 1 -- 172.21.15.17:0/848824093 shutdown_connections 2024-03-29T11:23:53.966 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.985+0000 7f1772cff640 1 -- 172.21.15.17:0/848824093 wait complete. 2024-03-29T11:23:53.966 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:23:53.986+0000 7f1772cff640 1 librados: shutdown 2024-03-29T11:23:53.986 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:00.793 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:00.942 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.961+0000 7f576dceb640 10 monclient: get_monmap_and_config 2024-03-29T11:24:00.943 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.961+0000 7f576dceb640 10 monclient: build_initial_monmap 2024-03-29T11:24:00.943 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.961+0000 7f576dceb640 10 monclient: monmap: 2024-03-29T11:24:00.943 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:24:00.943 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:00.943 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:24:00.962795+0000 2024-03-29T11:24:00.943 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:24:00.962795+0000 2024-03-29T11:24:00.943 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:00.943 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:00.943 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:24:00.943 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:24:00.943 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:24:00.943 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:00.944 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.962+0000 7f576dceb640 1 Processor -- start 2024-03-29T11:24:00.944 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.962+0000 7f576dceb640 1 -- start start 2024-03-29T11:24:00.944 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.962+0000 7f576dceb640 10 monclient: init 2024-03-29T11:24:00.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.963+0000 7f576dceb640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:00.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.963+0000 7f576dceb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:24:00.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.963+0000 7f576dceb640 10 monclient(hunting): picked mon.noname-c con 0x7f57680b2800 addr v1:172.21.15.94:6789/0 2024-03-29T11:24:00.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.963+0000 7f576dceb640 10 monclient(hunting): picked mon.noname-b con 0x7f57680b3220 addr v1:172.21.15.17:6790/0 2024-03-29T11:24:00.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.963+0000 7f576dceb640 10 monclient(hunting): picked mon.noname-a con 0x7f57681439c0 addr v1:172.21.15.17:6789/0 2024-03-29T11:24:00.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.963+0000 7f576dceb640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57680b3840 con 0x7f57681439c0 2024-03-29T11:24:00.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.963+0000 7f576dceb640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f576806f230 con 0x7f57680b2800 2024-03-29T11:24:00.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.963+0000 7f576dceb640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f576806ef50 con 0x7f57680b3220 2024-03-29T11:24:00.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.963+0000 7f576dceb640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:00.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.963+0000 7f576dceb640 10 monclient(hunting): authenticate will time out at 1147.168831s 2024-03-29T11:24:00.948 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.967+0000 7f5767fff640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f57680b3220 0x7f5768146710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:37985/0 (socket says 172.21.15.17:37985) 2024-03-29T11:24:00.948 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.967+0000 7f5767fff640 1 -- 172.21.15.17:0/3868614499 learned_addr learned my addr 172.21.15.17:0/3868614499 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:24:00.949 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.967+0000 7f57677fe640 1 -- 172.21.15.17:0/3868614499 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2412171944 0 0) 0x7f576806ef50 con 0x7f57680b3220 2024-03-29T11:24:00.949 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.967+0000 7f57677fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:00.949 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.967+0000 7f57677fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:00.949 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.968+0000 7f57677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:00.949 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.968+0000 7f57677fe640 10 monclient(hunting): my global_id is 4433 2024-03-29T11:24:00.952 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.968+0000 7f57677fe640 1 -- 172.21.15.17:0/3868614499 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f574c003520 con 0x7f57680b3220 2024-03-29T11:24:00.952 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.971+0000 7f57677fe640 1 -- 172.21.15.17:0/3868614499 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3555996972 0 0) 0x7f576806f230 con 0x7f57680b2800 2024-03-29T11:24:00.952 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.972+0000 7f57677fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:00.952 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.972+0000 7f57677fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:00.952 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.972+0000 7f57677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:00.953 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.972+0000 7f57677fe640 10 monclient(hunting): my global_id is 4423 2024-03-29T11:24:00.953 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.972+0000 7f57677fe640 1 -- 172.21.15.17:0/3868614499 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f576806ef50 con 0x7f57680b2800 2024-03-29T11:24:00.953 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.972+0000 7f57677fe640 1 -- 172.21.15.17:0/3868614499 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3026551450 0 0) 0x7f57680b3840 con 0x7f57681439c0 2024-03-29T11:24:00.953 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.972+0000 7f57677fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:00.953 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.972+0000 7f57677fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:00.953 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.972+0000 7f57677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:00.953 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.972+0000 7f57677fe640 10 monclient(hunting): my global_id is 4404 2024-03-29T11:24:00.953 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.972+0000 7f57677fe640 1 -- 172.21.15.17:0/3868614499 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f576806f230 con 0x7f57681439c0 2024-03-29T11:24:00.953 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.972+0000 7f57677fe640 1 -- 172.21.15.17:0/3868614499 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2694609610 0 0) 0x7f574c003520 con 0x7f57680b3220 2024-03-29T11:24:00.955 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.972+0000 7f57677fe640 1 -- 172.21.15.17:0/3868614499 >> v1:172.21.15.94:6789/0 conn(0x7f57680b2800 legacy=0x7f57681432b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:00.955 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.972+0000 7f57677fe640 1 -- 172.21.15.17:0/3868614499 >> v1:172.21.15.17:6789/0 conn(0x7f57681439c0 legacy=0x7f5768151c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:00.955 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.972+0000 7f57677fe640 10 monclient: _finish_hunting 0 2024-03-29T11:24:00.955 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.972+0000 7f57677fe640 1 monclient: found mon.noname-b 2024-03-29T11:24:00.956 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.972+0000 7f57677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:00.956 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.972+0000 7f57677fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.17:6790/0 2024-03-29T11:24:00.956 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.972+0000 7f57677fe640 1 -- 172.21.15.17:0/3868614499 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5768071d30 con 0x7f57680b3220 2024-03-29T11:24:00.956 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.972+0000 7f57677fe640 10 monclient: _finish_auth 0 2024-03-29T11:24:00.956 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.972+0000 7f57677fe640 10 monclient: _check_auth_tickets 2024-03-29T11:24:00.956 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.972+0000 7f57677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:00.956 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.972+0000 7f576dceb640 5 monclient: authenticate success, global_id 4433 2024-03-29T11:24:00.956 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.972+0000 7f576dceb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:00.956 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.972+0000 7f57677fe640 1 -- 172.21.15.17:0/3868614499 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f5754001690 con 0x7f57680b3220 2024-03-29T11:24:00.956 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.972+0000 7f57677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:00.956 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.972+0000 7f57677fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:00.956 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.972+0000 7f57677fe640 10 monclient: dump: 2024-03-29T11:24:00.956 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:24:00.956 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:00.957 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:00.957 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:00.957 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:00.957 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:00.957 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:24:00.957 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:24:00.957 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:24:00.957 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:00.957 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.972+0000 7f576dceb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:00.957 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.974+0000 7f57677fe640 1 -- 172.21.15.17:0/3868614499 <== mon.2 v1:172.21.15.17:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f57540017f0 con 0x7f57680b3220 2024-03-29T11:24:00.957 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.974+0000 7f57677fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:00.957 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.974+0000 7f576dceb640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:00.959 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.974+0000 7f576dceb640 4 set_mon_vals no callback set 2024-03-29T11:24:00.959 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.975+0000 7f576dceb640 10 monclient: shutdown 2024-03-29T11:24:00.959 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.975+0000 7f576dceb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:00.959 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.975+0000 7f576dceb640 1 -- 172.21.15.17:0/3868614499 >> v1:172.21.15.17:6790/0 conn(0x7f57680b3220 legacy=0x7f5768146710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:00.959 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.975+0000 7f576dceb640 1 -- 172.21.15.17:0/3868614499 shutdown_connections 2024-03-29T11:24:00.960 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.975+0000 7f576dceb640 1 -- 172.21.15.17:0/3868614499 >> 172.21.15.17:0/3868614499 conn(0x7f5768066a00 msgr2=0x7f5768066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:00.960 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.976+0000 7f576dceb640 1 -- 172.21.15.17:0/3868614499 shutdown_connections 2024-03-29T11:24:00.960 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.976+0000 7f576dceb640 1 -- 172.21.15.17:0/3868614499 wait complete. 2024-03-29T11:24:00.960 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.977+0000 7f576dceb640 10 monclient: build_initial_monmap 2024-03-29T11:24:00.960 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.977+0000 7f576dceb640 10 monclient: monmap: 2024-03-29T11:24:00.960 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:24:00.960 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:00.960 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:24:00.960 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:24:00.960 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:00.960 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:00.960 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:24:00.960 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:24:00.960 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:24:00.960 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:00.961 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.977+0000 7f576dceb640 1 librados: starting msgr at 2024-03-29T11:24:00.961 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.977+0000 7f576dceb640 1 librados: starting objecter 2024-03-29T11:24:00.962 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.977+0000 7f576dceb640 1 Processor -- start 2024-03-29T11:24:00.962 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.977+0000 7f576dceb640 1 -- start start 2024-03-29T11:24:00.962 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.977+0000 7f576dceb640 1 librados: setting wanted keys 2024-03-29T11:24:00.962 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.977+0000 7f576dceb640 1 librados: calling monclient init 2024-03-29T11:24:00.962 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.977+0000 7f576dceb640 10 monclient: init 2024-03-29T11:24:00.962 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.978+0000 7f576dceb640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:00.962 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.978+0000 7f576dceb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:24:00.962 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.978+0000 7f576dceb640 10 monclient(hunting): picked mon.noname-b con 0x7f57680b2800 addr v1:172.21.15.94:6789/0 2024-03-29T11:24:00.962 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.978+0000 7f576dceb640 10 monclient(hunting): picked mon.noname-c con 0x7f57681439c0 addr v1:172.21.15.17:6790/0 2024-03-29T11:24:00.962 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.978+0000 7f576dceb640 10 monclient(hunting): picked mon.noname-a con 0x7f57681cf650 addr v1:172.21.15.17:6789/0 2024-03-29T11:24:00.962 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.978+0000 7f576dceb640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5768071810 con 0x7f57681cf650 2024-03-29T11:24:00.962 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.978+0000 7f576dceb640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57681cfae0 con 0x7f57680b2800 2024-03-29T11:24:00.962 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.978+0000 7f576dceb640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57681cf150 con 0x7f57681439c0 2024-03-29T11:24:00.963 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.978+0000 7f576dceb640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:00.963 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.978+0000 7f576dceb640 10 monclient(hunting): authenticate will time out at 1147.183803s 2024-03-29T11:24:00.964 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.983+0000 7f576cce9640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f57680b2800 0x7f57681d0fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:55433/0 (socket says 172.21.15.17:55433) 2024-03-29T11:24:00.964 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.983+0000 7f576cce9640 1 -- 172.21.15.17:0/1107075986 learned_addr learned my addr 172.21.15.17:0/1107075986 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:24:00.966 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.985+0000 7f57657fa640 10 client.?.objecter ms_handle_connect 0x7f57680b2800 2024-03-29T11:24:00.966 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.985+0000 7f57657fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:00.966 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.985+0000 7f57657fa640 10 client.?.objecter ms_handle_connect 0x7f57681cf650 2024-03-29T11:24:00.966 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.985+0000 7f57657fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:00.966 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.985+0000 7f57657fa640 1 -- 172.21.15.17:0/1107075986 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2085791085 0 0) 0x7f57681cfae0 con 0x7f57680b2800 2024-03-29T11:24:00.966 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.985+0000 7f57657fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:00.966 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.985+0000 7f57657fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:00.966 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.985+0000 7f57657fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:00.966 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.985+0000 7f57657fa640 10 monclient(hunting): my global_id is 4426 2024-03-29T11:24:00.967 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.985+0000 7f57657fa640 1 -- 172.21.15.17:0/1107075986 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5750004eb0 con 0x7f57680b2800 2024-03-29T11:24:00.969 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.987+0000 7f57657fa640 10 client.?.objecter ms_handle_connect 0x7f57681439c0 2024-03-29T11:24:00.969 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.987+0000 7f57657fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:00.969 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.987+0000 7f57657fa640 1 -- 172.21.15.17:0/1107075986 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 385689226 0 0) 0x7f57681cf150 con 0x7f57681439c0 2024-03-29T11:24:00.969 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.987+0000 7f57657fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:00.969 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.987+0000 7f57657fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:00.969 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.987+0000 7f57657fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:00.969 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.987+0000 7f57657fa640 10 monclient(hunting): my global_id is 4436 2024-03-29T11:24:00.969 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.987+0000 7f57657fa640 1 -- 172.21.15.17:0/1107075986 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57681cfae0 con 0x7f57681439c0 2024-03-29T11:24:00.969 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.987+0000 7f57657fa640 1 -- 172.21.15.17:0/1107075986 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 314673396 0 0) 0x7f5750004eb0 con 0x7f57680b2800 2024-03-29T11:24:00.970 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.987+0000 7f57657fa640 1 -- 172.21.15.17:0/1107075986 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57681cf150 con 0x7f57680b2800 2024-03-29T11:24:00.970 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.987+0000 7f57657fa640 1 -- 172.21.15.17:0/1107075986 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f5760001dd0 con 0x7f57680b2800 2024-03-29T11:24:00.970 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.987+0000 7f57657fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:00.970 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.987+0000 7f57657fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:00.970 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.987+0000 7f57657fa640 10 monclient(hunting): dump: 2024-03-29T11:24:00.970 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:24:00.970 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:00.970 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:00.970 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:00.970 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:00.970 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:00.970 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:24:00.970 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:24:00.970 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:24:00.970 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:00.971 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.987+0000 7f57657fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:00.971 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.987+0000 7f57657fa640 1 -- 172.21.15.17:0/1107075986 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2395646356 0 0) 0x7f5768071810 con 0x7f57681cf650 2024-03-29T11:24:00.975 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.987+0000 7f57657fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:00.975 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.987+0000 7f57657fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:00.975 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.987+0000 7f57657fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:00.975 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.987+0000 7f57657fa640 10 monclient(hunting): my global_id is 4407 2024-03-29T11:24:00.975 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.987+0000 7f57657fa640 1 -- 172.21.15.17:0/1107075986 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5750004eb0 con 0x7f57681cf650 2024-03-29T11:24:00.975 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.987+0000 7f57657fa640 1 -- 172.21.15.17:0/1107075986 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3459048714 0 0) 0x7f57681cfae0 con 0x7f57681439c0 2024-03-29T11:24:00.975 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.987+0000 7f57657fa640 1 -- 172.21.15.17:0/1107075986 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5768071810 con 0x7f57681439c0 2024-03-29T11:24:00.975 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.987+0000 7f57657fa640 1 -- 172.21.15.17:0/1107075986 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f57540047e0 con 0x7f57681439c0 2024-03-29T11:24:00.975 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.987+0000 7f57657fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:00.975 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.987+0000 7f57657fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:00.975 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.987+0000 7f57657fa640 10 monclient(hunting): dump: 2024-03-29T11:24:00.975 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:24:00.976 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:00.976 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:00.976 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:00.976 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:00.976 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:00.976 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:24:00.976 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:24:00.976 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:24:00.976 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:00.976 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.989+0000 7f57657fa640 1 -- 172.21.15.17:0/1107075986 <== mon.1 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2119793815 0 0) 0x7f57681cf150 con 0x7f57680b2800 2024-03-29T11:24:00.976 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.989+0000 7f57657fa640 1 -- 172.21.15.17:0/1107075986 >> v1:172.21.15.17:6790/0 conn(0x7f57681439c0 legacy=0x7f57681ce6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:00.976 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.989+0000 7f57657fa640 1 -- 172.21.15.17:0/1107075986 >> v1:172.21.15.17:6789/0 conn(0x7f57681cf650 legacy=0x7f57681db340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:00.976 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.989+0000 7f57657fa640 10 monclient: _finish_hunting 0 2024-03-29T11:24:00.976 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.989+0000 7f57657fa640 1 monclient: found mon.b 2024-03-29T11:24:00.976 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.989+0000 7f57657fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:00.977 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.989+0000 7f57657fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:24:00.977 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.989+0000 7f57657fa640 1 -- 172.21.15.17:0/1107075986 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5768146100 con 0x7f57680b2800 2024-03-29T11:24:00.977 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.989+0000 7f57657fa640 10 monclient: _finish_auth 0 2024-03-29T11:24:00.977 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.989+0000 7f57657fa640 10 monclient: _check_auth_tickets 2024-03-29T11:24:00.977 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.989+0000 7f57657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:00.977 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.989+0000 7f576dceb640 5 monclient: authenticate success, global_id 4426 2024-03-29T11:24:00.977 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.989+0000 7f576dceb640 10 monclient: _renew_subs 2024-03-29T11:24:00.977 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.989+0000 7f576dceb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:24:00.977 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.989+0000 7f576dceb640 1 -- 172.21.15.17:0/1107075986 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5768070d80 con 0x7f57680b2800 2024-03-29T11:24:00.977 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.989+0000 7f576dceb640 10 client.4426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:00.977 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.989+0000 7f576dceb640 10 monclient: _renew_subs 2024-03-29T11:24:00.977 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.989+0000 7f576dceb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:24:00.977 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.989+0000 7f576dceb640 1 -- 172.21.15.17:0/1107075986 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f576813db30 con 0x7f57680b2800 2024-03-29T11:24:00.978 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.989+0000 7f576dceb640 1 librados: init done 2024-03-29T11:24:00.978 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.992+0000 7f57657fa640 1 -- 172.21.15.17:0/1107075986 <== mon.1 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5760002050 con 0x7f57680b2800 2024-03-29T11:24:00.978 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.992+0000 7f57657fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:00.978 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.992+0000 7f57657fa640 1 -- 172.21.15.17:0/1107075986 <== mon.1 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f5760001a30 con 0x7f57680b2800 2024-03-29T11:24:00.978 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.992+0000 7f57657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:00.978 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.992+0000 7f57657fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:00.978 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.992+0000 7f57657fa640 10 monclient: dump: 2024-03-29T11:24:00.978 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:24:00.978 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:00.978 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:00.978 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:00.978 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:00.978 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:00.978 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:24:00.979 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:24:00.979 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:24:00.979 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:00.979 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.992+0000 7f576dceb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:00.979 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.992+0000 7f5766ffd640 4 set_mon_vals no callback set 2024-03-29T11:24:00.979 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.993+0000 7f576dceb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:00.979 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.993+0000 7f576dceb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:24:00.979 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.993+0000 7f576dceb640 1 -- 172.21.15.17:0/1107075986 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57680b3ed0 con 0x7f57680b2800 2024-03-29T11:24:00.980 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.994+0000 7f57657fa640 1 -- 172.21.15.17:0/1107075986 <== mon.1 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f5760003e80 con 0x7f57680b2800 2024-03-29T11:24:00.980 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.995+0000 7f57657fa640 1 -- 172.21.15.17:0/1107075986 <== mon.1 v1:172.21.15.94:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f5760084de0 con 0x7f57680b2800 2024-03-29T11:24:00.980 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.995+0000 7f57657fa640 10 client.4426.objecter ms_dispatch 0x7f5768000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:00.980 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.995+0000 7f57657fa640 3 client.4426.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:00.980 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.995+0000 7f57657fa640 3 client.4426.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:00.980 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.995+0000 7f57657fa640 20 client.4426.objecter dump_active .. 0 homeless 2024-03-29T11:24:00.980 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.995+0000 7f5767fff640 10 monclient: get_auth_request con 0x7f575006dfe0 auth_method 0 2024-03-29T11:24:00.980 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:00.996+0000 7f57657fa640 10 client.4426.objecter ms_handle_connect 0x7f575006dfe0 2024-03-29T11:24:00.983 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:01.002+0000 7f57657fa640 1 -- 172.21.15.17:0/1107075986 <== mon.1 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f57600b48b0 con 0x7f57680b2800 2024-03-29T11:24:00.983 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:01.002+0000 7f57657fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:00.983 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:01.002+0000 7f57657fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:01.315 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:01.334+0000 7f576dceb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:01.315 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:01.334+0000 7f576dceb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:01.315 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:01.334+0000 7f576dceb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:24:01.315 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:01.334+0000 7f576dceb640 1 -- 172.21.15.17:0/1107075986 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f576804f730 con 0x7f57680b2800 2024-03-29T11:24:01.317 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:24:01.317 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:01.317 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:01.336+0000 7f57657fa640 1 -- 172.21.15.17:0/1107075986 <== mon.1 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f57600506c0 con 0x7f57680b2800 2024-03-29T11:24:01.317 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:01.336+0000 7f57657fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:01.317 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:01.336+0000 7f57657fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:01.319 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:01.339+0000 7f576dceb640 10 librados: watch_flush enter 2024-03-29T11:24:01.320 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:01.339+0000 7f576dceb640 10 librados: watch_flush exit 2024-03-29T11:24:01.320 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:01.339+0000 7f576dceb640 20 client.4426.objecter shutdown clearing up homeless session... 2024-03-29T11:24:01.320 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:01.339+0000 7f576dceb640 10 client.4426.objecter successfully canceled tick 2024-03-29T11:24:01.320 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:01.340+0000 7f576dceb640 1 -- 172.21.15.17:0/1107075986 >> v1:172.21.15.17:6816/3236185052 conn(0x7f575006dfe0 legacy=0x7f57500704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.321 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:01.340+0000 7f576dceb640 10 monclient: shutdown 2024-03-29T11:24:01.321 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:01.340+0000 7f576dceb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:01.321 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:01.340+0000 7f576dceb640 1 -- 172.21.15.17:0/1107075986 >> v1:172.21.15.94:6789/0 conn(0x7f57680b2800 legacy=0x7f57681d0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.321 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:01.340+0000 7f576dceb640 1 -- 172.21.15.17:0/1107075986 shutdown_connections 2024-03-29T11:24:01.321 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:01.340+0000 7f576dceb640 1 -- 172.21.15.17:0/1107075986 >> 172.21.15.17:0/1107075986 conn(0x7f5768066a00 msgr2=0x7f5768069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:01.322 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:01.342+0000 7f576dceb640 1 -- 172.21.15.17:0/1107075986 shutdown_connections 2024-03-29T11:24:01.322 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:01.342+0000 7f576dceb640 1 -- 172.21.15.17:0/1107075986 wait complete. 2024-03-29T11:24:01.323 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:01.342+0000 7f576dceb640 1 librados: shutdown 2024-03-29T11:24:01.343 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:08.149 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:08.298 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.316+0000 7f84c1d84640 10 monclient: get_monmap_and_config 2024-03-29T11:24:08.298 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.316+0000 7f84c1d84640 10 monclient: build_initial_monmap 2024-03-29T11:24:08.298 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.316+0000 7f84c1d84640 10 monclient: monmap: 2024-03-29T11:24:08.298 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:24:08.298 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:08.298 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:24:08.317623+0000 2024-03-29T11:24:08.298 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:24:08.317623+0000 2024-03-29T11:24:08.298 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:08.298 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:08.298 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:24:08.299 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:24:08.299 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:24:08.299 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:08.300 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.317+0000 7f84c1d84640 1 Processor -- start 2024-03-29T11:24:08.300 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.317+0000 7f84c1d84640 1 -- start start 2024-03-29T11:24:08.300 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.317+0000 7f84c1d84640 10 monclient: init 2024-03-29T11:24:08.300 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.318+0000 7f84c1d84640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:08.300 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.318+0000 7f84c1d84640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:08.300 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.318+0000 7f84c1d84640 10 monclient(hunting): picked mon.noname-a con 0x7f84bc0a9750 addr v1:172.21.15.17:6789/0 2024-03-29T11:24:08.300 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.318+0000 7f84c1d84640 10 monclient(hunting): picked mon.noname-c con 0x7f84bc0aad20 addr v1:172.21.15.94:6789/0 2024-03-29T11:24:08.303 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.321+0000 7f84c1d84640 10 monclient(hunting): picked mon.noname-b con 0x7f84bc0a9b30 addr v1:172.21.15.17:6790/0 2024-03-29T11:24:08.303 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.321+0000 7f84c1d84640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84bc0af4d0 con 0x7f84bc0a9750 2024-03-29T11:24:08.303 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.321+0000 7f84c1d84640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84bc0af7b0 con 0x7f84bc0aad20 2024-03-29T11:24:08.303 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.321+0000 7f84c1d84640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84bc0acab0 con 0x7f84bc0a9b30 2024-03-29T11:24:08.303 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.321+0000 7f84c1d84640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:08.303 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.321+0000 7f84c1d84640 10 monclient(hunting): authenticate will time out at 1154.526706s 2024-03-29T11:24:08.303 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.322+0000 7f84c1583640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f84bc0a9b30 0x7f84bc149930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:51955/0 (socket says 172.21.15.17:51955) 2024-03-29T11:24:08.303 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.322+0000 7f84c1583640 1 -- 172.21.15.17:0/3431809021 learned_addr learned my addr 172.21.15.17:0/3431809021 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:24:08.310 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.324+0000 7f84bb7fe640 1 -- 172.21.15.17:0/3431809021 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1998405319 0 0) 0x7f84bc0af4d0 con 0x7f84bc0a9750 2024-03-29T11:24:08.310 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.324+0000 7f84bb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:08.310 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.324+0000 7f84bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:08.310 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.324+0000 7f84bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:08.310 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.324+0000 7f84bb7fe640 10 monclient(hunting): my global_id is 4410 2024-03-29T11:24:08.310 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.324+0000 7f84bb7fe640 1 -- 172.21.15.17:0/3431809021 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84a4003520 con 0x7f84bc0a9750 2024-03-29T11:24:08.310 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.325+0000 7f84bb7fe640 1 -- 172.21.15.17:0/3431809021 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1877333648 0 0) 0x7f84bc0af7b0 con 0x7f84bc0aad20 2024-03-29T11:24:08.310 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.325+0000 7f84bb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:08.310 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.325+0000 7f84bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:08.310 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.325+0000 7f84bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:08.310 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.325+0000 7f84bb7fe640 10 monclient(hunting): my global_id is 4429 2024-03-29T11:24:08.311 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.325+0000 7f84bb7fe640 1 -- 172.21.15.17:0/3431809021 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84bc0af4d0 con 0x7f84bc0aad20 2024-03-29T11:24:08.311 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.325+0000 7f84bb7fe640 1 -- 172.21.15.17:0/3431809021 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3728962242 0 0) 0x7f84bc0af4d0 con 0x7f84bc0aad20 2024-03-29T11:24:08.311 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.325+0000 7f84bb7fe640 1 -- 172.21.15.17:0/3431809021 >> v1:172.21.15.17:6790/0 conn(0x7f84bc0a9b30 legacy=0x7f84bc149930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:08.311 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.325+0000 7f84bb7fe640 1 -- 172.21.15.17:0/3431809021 >> v1:172.21.15.17:6789/0 conn(0x7f84bc0a9750 legacy=0x7f84bc0ac3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:08.311 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.325+0000 7f84bb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:24:08.311 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.325+0000 7f84bb7fe640 1 monclient: found mon.noname-c 2024-03-29T11:24:08.311 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.325+0000 7f84bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:08.311 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.325+0000 7f84bb7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6789/0 2024-03-29T11:24:08.311 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.325+0000 7f84bb7fe640 1 -- 172.21.15.17:0/3431809021 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84bc0aba30 con 0x7f84bc0aad20 2024-03-29T11:24:08.311 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.325+0000 7f84bb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:24:08.311 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.325+0000 7f84bb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:24:08.311 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.325+0000 7f84bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:08.311 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.325+0000 7f84bb7fe640 1 -- 172.21.15.17:0/3431809021 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f84b0001670 con 0x7f84bc0aad20 2024-03-29T11:24:08.311 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.325+0000 7f84bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:08.312 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.325+0000 7f84bb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:08.312 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.325+0000 7f84bb7fe640 10 monclient: dump: 2024-03-29T11:24:08.312 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:24:08.312 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:08.312 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:08.312 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:08.312 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:08.312 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:08.312 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:24:08.312 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:24:08.312 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:24:08.312 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:08.312 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.326+0000 7f84bb7fe640 1 -- 172.21.15.17:0/3431809021 <== mon.1 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f84b00017d0 con 0x7f84bc0aad20 2024-03-29T11:24:08.312 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.326+0000 7f84bb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:08.312 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.326+0000 7f84bb7fe640 1 -- 172.21.15.17:0/3431809021 <== mon.1 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f84b0001ba0 con 0x7f84bc0aad20 2024-03-29T11:24:08.312 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.326+0000 7f84bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:08.313 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.326+0000 7f84bb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:08.313 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.326+0000 7f84bb7fe640 10 monclient: dump: 2024-03-29T11:24:08.313 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:24:08.313 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:08.313 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:08.313 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:08.313 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:08.313 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:08.313 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:24:08.313 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:24:08.313 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:24:08.313 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:08.313 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.327+0000 7f84c1d84640 5 monclient: authenticate success, global_id 4429 2024-03-29T11:24:08.313 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.327+0000 7f84c1d84640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:08.314 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.327+0000 7f84c1d84640 4 set_mon_vals no callback set 2024-03-29T11:24:08.314 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.328+0000 7f84c1d84640 10 monclient: shutdown 2024-03-29T11:24:08.314 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.328+0000 7f84c1d84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:08.314 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.328+0000 7f84c1d84640 1 -- 172.21.15.17:0/3431809021 >> v1:172.21.15.94:6789/0 conn(0x7f84bc0aad20 legacy=0x7f84bc141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:08.314 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.328+0000 7f84c1d84640 1 -- 172.21.15.17:0/3431809021 shutdown_connections 2024-03-29T11:24:08.314 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.328+0000 7f84c1d84640 1 -- 172.21.15.17:0/3431809021 >> 172.21.15.17:0/3431809021 conn(0x7f84bc0673e0 msgr2=0x7f84bc0b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:08.315 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.330+0000 7f84c1d84640 1 -- 172.21.15.17:0/3431809021 shutdown_connections 2024-03-29T11:24:08.316 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.330+0000 7f84c1d84640 1 -- 172.21.15.17:0/3431809021 wait complete. 2024-03-29T11:24:08.316 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.331+0000 7f84c1d84640 10 monclient: build_initial_monmap 2024-03-29T11:24:08.316 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.331+0000 7f84c1d84640 10 monclient: monmap: 2024-03-29T11:24:08.316 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:24:08.316 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:08.316 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:24:08.316 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:24:08.316 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:08.316 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:08.316 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:24:08.316 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:24:08.316 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:24:08.316 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:08.317 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.331+0000 7f84c1d84640 1 librados: starting msgr at 2024-03-29T11:24:08.317 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.331+0000 7f84c1d84640 1 librados: starting objecter 2024-03-29T11:24:08.317 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.331+0000 7f84c1d84640 1 Processor -- start 2024-03-29T11:24:08.317 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.332+0000 7f84c1d84640 1 -- start start 2024-03-29T11:24:08.317 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.332+0000 7f84c1d84640 1 librados: setting wanted keys 2024-03-29T11:24:08.317 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.332+0000 7f84c1d84640 1 librados: calling monclient init 2024-03-29T11:24:08.317 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.332+0000 7f84c1d84640 10 monclient: init 2024-03-29T11:24:08.317 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.333+0000 7f84c1d84640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:08.317 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.333+0000 7f84c1d84640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:08.317 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.333+0000 7f84c1d84640 10 monclient(hunting): picked mon.noname-a con 0x7f84bc0a9750 addr v1:172.21.15.17:6789/0 2024-03-29T11:24:08.317 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.333+0000 7f84c1d84640 10 monclient(hunting): picked mon.noname-b con 0x7f84bc0a9b30 addr v1:172.21.15.94:6789/0 2024-03-29T11:24:08.318 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.336+0000 7f84c1d84640 10 monclient(hunting): picked mon.noname-c con 0x7f84bc0aad20 addr v1:172.21.15.17:6790/0 2024-03-29T11:24:08.318 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.337+0000 7f84c1d84640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84bc12cf00 con 0x7f84bc0a9750 2024-03-29T11:24:08.318 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.337+0000 7f84c1d84640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84bc12d1e0 con 0x7f84bc0a9b30 2024-03-29T11:24:08.319 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.337+0000 7f84c1d84640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84bc071a10 con 0x7f84bc0aad20 2024-03-29T11:24:08.319 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.337+0000 7f84c1d84640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:08.319 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.337+0000 7f84c1d84640 10 monclient(hunting): authenticate will time out at 1154.542312s 2024-03-29T11:24:08.323 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.341+0000 7f84c1583640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f84bc0aad20 0x7f84bc071300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:52035/0 (socket says 172.21.15.17:52035) 2024-03-29T11:24:08.323 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.341+0000 7f84c1583640 1 -- 172.21.15.17:0/3678356044 learned_addr learned my addr 172.21.15.17:0/3678356044 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:24:08.323 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.341+0000 7f84b97fa640 10 client.?.objecter ms_handle_connect 0x7f84bc0aad20 2024-03-29T11:24:08.323 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.341+0000 7f84b97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:08.324 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.341+0000 7f84b97fa640 1 -- 172.21.15.17:0/3678356044 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3081209664 0 0) 0x7f84bc071a10 con 0x7f84bc0aad20 2024-03-29T11:24:08.324 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.341+0000 7f84b97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:08.324 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.341+0000 7f84b97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:08.324 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.341+0000 7f84b97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:08.324 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.341+0000 7f84b97fa640 10 monclient(hunting): my global_id is 4439 2024-03-29T11:24:08.324 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.341+0000 7f84b97fa640 1 -- 172.21.15.17:0/3678356044 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84a8004eb0 con 0x7f84bc0aad20 2024-03-29T11:24:08.324 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.342+0000 7f84b97fa640 10 client.?.objecter ms_handle_connect 0x7f84bc0a9750 2024-03-29T11:24:08.324 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.342+0000 7f84b97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:08.324 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.342+0000 7f84b97fa640 1 -- 172.21.15.17:0/3678356044 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2911996246 0 0) 0x7f84bc12cf00 con 0x7f84bc0a9750 2024-03-29T11:24:08.324 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.342+0000 7f84b97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:08.325 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.342+0000 7f84b97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:08.325 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.342+0000 7f84b97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:08.325 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.342+0000 7f84b97fa640 10 monclient(hunting): my global_id is 4413 2024-03-29T11:24:08.325 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.342+0000 7f84b97fa640 1 -- 172.21.15.17:0/3678356044 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84bc071a10 con 0x7f84bc0a9750 2024-03-29T11:24:08.328 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.343+0000 7f84b97fa640 1 -- 172.21.15.17:0/3678356044 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4228863740 0 0) 0x7f84bc071a10 con 0x7f84bc0a9750 2024-03-29T11:24:08.329 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.343+0000 7f84b97fa640 1 -- 172.21.15.17:0/3678356044 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f84a8005490 con 0x7f84bc0a9750 2024-03-29T11:24:08.329 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.343+0000 7f84b97fa640 1 -- 172.21.15.17:0/3678356044 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f84ac001950 con 0x7f84bc0a9750 2024-03-29T11:24:08.329 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.343+0000 7f84b97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:08.329 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.343+0000 7f84b97fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:08.329 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.343+0000 7f84b97fa640 10 monclient(hunting): dump: 2024-03-29T11:24:08.329 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:24:08.329 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:08.329 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:08.329 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:08.329 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:08.329 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:08.329 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:24:08.329 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:24:08.330 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:24:08.330 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:08.330 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.343+0000 7f84b97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:08.330 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.344+0000 7f84b97fa640 1 -- 172.21.15.17:0/3678356044 <== mon.0 v1:172.21.15.17:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1333380845 0 0) 0x7f84a8005490 con 0x7f84bc0a9750 2024-03-29T11:24:08.330 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.344+0000 7f84b97fa640 1 -- 172.21.15.17:0/3678356044 >> v1:172.21.15.17:6790/0 conn(0x7f84bc0aad20 legacy=0x7f84bc071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:08.330 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.344+0000 7f84b97fa640 1 -- 172.21.15.17:0/3678356044 >> v1:172.21.15.94:6789/0 conn(0x7f84bc0a9b30 legacy=0x7f84bc070bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:08.330 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.344+0000 7f84b97fa640 10 monclient: _finish_hunting 0 2024-03-29T11:24:08.330 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.344+0000 7f84b97fa640 1 monclient: found mon.a 2024-03-29T11:24:08.330 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.344+0000 7f84b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:08.330 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.344+0000 7f84b97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:24:08.330 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.344+0000 7f84b97fa640 1 -- 172.21.15.17:0/3678356044 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84bc1da7f0 con 0x7f84bc0a9750 2024-03-29T11:24:08.330 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.344+0000 7f84b97fa640 10 monclient: _finish_auth 0 2024-03-29T11:24:08.330 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.344+0000 7f84b97fa640 10 monclient: _check_auth_tickets 2024-03-29T11:24:08.331 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.344+0000 7f84b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:08.331 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.344+0000 7f84b97fa640 10 client.?.objecter ms_handle_connect 0x7f84bc0a9b30 2024-03-29T11:24:08.331 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.344+0000 7f84b97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:08.331 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.344+0000 7f84c1d84640 5 monclient: authenticate success, global_id 4413 2024-03-29T11:24:08.331 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.344+0000 7f84c1d84640 10 monclient: _renew_subs 2024-03-29T11:24:08.331 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.344+0000 7f84c1d84640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:24:08.331 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.344+0000 7f84c1d84640 1 -- 172.21.15.17:0/3678356044 --> v1:172.21.15.17:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84bc1da9e0 con 0x7f84bc0a9750 2024-03-29T11:24:08.331 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.344+0000 7f84c1d84640 10 client.4413.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:08.331 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.344+0000 7f84c1d84640 10 monclient: _renew_subs 2024-03-29T11:24:08.331 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.344+0000 7f84c1d84640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:24:08.332 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.344+0000 7f84c1d84640 1 -- 172.21.15.17:0/3678356044 --> v1:172.21.15.17:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f84bc1e72c0 con 0x7f84bc0a9750 2024-03-29T11:24:08.333 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.344+0000 7f84c1d84640 1 librados: init done 2024-03-29T11:24:08.333 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.347+0000 7f84b97fa640 1 -- 172.21.15.17:0/3678356044 <== mon.0 v1:172.21.15.17:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f84ac001d60 con 0x7f84bc0a9750 2024-03-29T11:24:08.333 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.347+0000 7f84b97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:08.333 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.347+0000 7f84baffd640 4 set_mon_vals no callback set 2024-03-29T11:24:08.333 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.348+0000 7f84b97fa640 1 -- 172.21.15.17:0/3678356044 <== mon.0 v1:172.21.15.17:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f84ac0037b0 con 0x7f84bc0a9750 2024-03-29T11:24:08.333 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.348+0000 7f84b97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:08.333 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.348+0000 7f84b97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:08.333 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.348+0000 7f84b97fa640 10 monclient: dump: 2024-03-29T11:24:08.333 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:24:08.333 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:08.333 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:08.333 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:08.333 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:08.333 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:08.334 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:24:08.334 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:24:08.334 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:24:08.334 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:08.334 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.350+0000 7f84b97fa640 1 -- 172.21.15.17:0/3678356044 <== mon.0 v1:172.21.15.17:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f84ac003990 con 0x7f84bc0a9750 2024-03-29T11:24:08.334 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.350+0000 7f84b97fa640 1 -- 172.21.15.17:0/3678356044 <== mon.0 v1:172.21.15.17:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f84ac083710 con 0x7f84bc0a9750 2024-03-29T11:24:08.334 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.350+0000 7f84b97fa640 10 client.4413.objecter ms_dispatch 0x7f84bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:08.334 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.350+0000 7f84b97fa640 3 client.4413.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:08.334 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.350+0000 7f84b97fa640 3 client.4413.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:08.334 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.350+0000 7f84b97fa640 20 client.4413.objecter dump_active .. 0 homeless 2024-03-29T11:24:08.335 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.351+0000 7f84a2ffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:08.335 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.351+0000 7f84a2ffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:08.335 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.351+0000 7f84a2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:24:08.335 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.351+0000 7f84a2ffd640 1 -- 172.21.15.17:0/3678356044 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f849c005080 con 0x7f84bc0a9750 2024-03-29T11:24:08.335 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.351+0000 7f84bbfff640 10 monclient: get_auth_request con 0x7f84a806de20 auth_method 0 2024-03-29T11:24:08.335 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.352+0000 7f84b97fa640 10 client.4413.objecter ms_handle_connect 0x7f84a806de20 2024-03-29T11:24:08.340 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.360+0000 7f84b97fa640 1 -- 172.21.15.17:0/3678356044 <== mon.0 v1:172.21.15.17:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f84ac04eff0 con 0x7f84bc0a9750 2024-03-29T11:24:08.340 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.360+0000 7f84b97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:08.340 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.360+0000 7f84b97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:08.670 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.689+0000 7f84a2ffd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:08.670 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.689+0000 7f84a2ffd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:08.671 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.689+0000 7f84a2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:24:08.671 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.689+0000 7f84a2ffd640 1 -- 172.21.15.17:0/3678356044 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f849c005330 con 0x7f84bc0a9750 2024-03-29T11:24:08.671 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.690+0000 7f84b97fa640 1 -- 172.21.15.17:0/3678356044 <== mon.0 v1:172.21.15.17:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f84ac050f20 con 0x7f84bc0a9750 2024-03-29T11:24:08.671 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.690+0000 7f84b97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:08.671 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.690+0000 7f84b97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:08.673 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:24:08.674 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:08.676 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.695+0000 7f84a2ffd640 10 librados: watch_flush enter 2024-03-29T11:24:08.677 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.696+0000 7f84a2ffd640 10 librados: watch_flush exit 2024-03-29T11:24:08.677 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.696+0000 7f84a2ffd640 20 client.4413.objecter shutdown clearing up homeless session... 2024-03-29T11:24:08.677 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.696+0000 7f84a2ffd640 10 client.4413.objecter successfully canceled tick 2024-03-29T11:24:08.680 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.699+0000 7f84a2ffd640 1 -- 172.21.15.17:0/3678356044 >> v1:172.21.15.17:6816/3236185052 conn(0x7f84a806de20 legacy=0x7f84a80702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:08.680 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.699+0000 7f84a2ffd640 10 monclient: shutdown 2024-03-29T11:24:08.680 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.699+0000 7f84a2ffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:08.680 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.699+0000 7f84a2ffd640 1 -- 172.21.15.17:0/3678356044 >> v1:172.21.15.17:6789/0 conn(0x7f84bc0a9750 legacy=0x7f84bc1e76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:08.680 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.700+0000 7f84a2ffd640 1 -- 172.21.15.17:0/3678356044 shutdown_connections 2024-03-29T11:24:08.681 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.700+0000 7f84a2ffd640 1 -- 172.21.15.17:0/3678356044 >> 172.21.15.17:0/3678356044 conn(0x7f84bc0673e0 msgr2=0x7f84bc0ac340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:08.682 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.701+0000 7f84a2ffd640 1 -- 172.21.15.17:0/3678356044 shutdown_connections 2024-03-29T11:24:08.682 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.702+0000 7f84a2ffd640 1 -- 172.21.15.17:0/3678356044 wait complete. 2024-03-29T11:24:08.684 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:08.703+0000 7f84a2ffd640 1 librados: shutdown 2024-03-29T11:24:08.702 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:15.509 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:15.660 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.678+0000 7fca7cab3640 10 monclient: get_monmap_and_config 2024-03-29T11:24:15.660 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.678+0000 7fca7cab3640 10 monclient: build_initial_monmap 2024-03-29T11:24:15.661 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.678+0000 7fca7cab3640 10 monclient: monmap: 2024-03-29T11:24:15.661 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:24:15.661 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:15.661 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:24:15.680358+0000 2024-03-29T11:24:15.661 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:24:15.680358+0000 2024-03-29T11:24:15.661 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:15.661 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:15.661 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:24:15.661 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:24:15.661 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:24:15.661 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:15.662 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.679+0000 7fca7cab3640 1 Processor -- start 2024-03-29T11:24:15.662 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.679+0000 7fca7cab3640 1 -- start start 2024-03-29T11:24:15.662 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.679+0000 7fca7cab3640 10 monclient: init 2024-03-29T11:24:15.663 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.680+0000 7fca7cab3640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:15.663 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.680+0000 7fca7cab3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:24:15.663 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.680+0000 7fca7cab3640 10 monclient(hunting): picked mon.noname-c con 0x7fca780ae490 addr v1:172.21.15.94:6789/0 2024-03-29T11:24:15.663 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.680+0000 7fca7cab3640 10 monclient(hunting): picked mon.noname-b con 0x7fca780aef70 addr v1:172.21.15.17:6790/0 2024-03-29T11:24:15.663 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.680+0000 7fca7cab3640 10 monclient(hunting): picked mon.noname-a con 0x7fca780727a0 addr v1:172.21.15.17:6789/0 2024-03-29T11:24:15.663 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.680+0000 7fca7cab3640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca780b0020 con 0x7fca780727a0 2024-03-29T11:24:15.663 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.680+0000 7fca7cab3640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca7806d960 con 0x7fca780ae490 2024-03-29T11:24:15.663 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.680+0000 7fca7cab3640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca780ad310 con 0x7fca780aef70 2024-03-29T11:24:15.663 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.680+0000 7fca7cab3640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:15.664 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.680+0000 7fca7cab3640 10 monclient(hunting): authenticate will time out at 1161.886813s 2024-03-29T11:24:15.664 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.681+0000 7fca76ffd640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7fca780aef70 0x7fca78072090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:43060/0 (socket says 172.21.15.17:43060) 2024-03-29T11:24:15.664 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.681+0000 7fca76ffd640 1 -- 172.21.15.17:0/10755211 learned_addr learned my addr 172.21.15.17:0/10755211 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:24:15.664 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.681+0000 7fca767fc640 1 -- 172.21.15.17:0/10755211 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3762871670 0 0) 0x7fca7806d960 con 0x7fca780ae490 2024-03-29T11:24:15.664 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.681+0000 7fca767fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:15.664 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.681+0000 7fca767fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:15.665 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.681+0000 7fca767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:15.665 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.681+0000 7fca767fc640 10 monclient(hunting): my global_id is 4435 2024-03-29T11:24:15.665 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.681+0000 7fca767fc640 1 -- 172.21.15.17:0/10755211 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca60003520 con 0x7fca780ae490 2024-03-29T11:24:15.666 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.682+0000 7fca767fc640 1 -- 172.21.15.17:0/10755211 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3761360841 0 0) 0x7fca780ad310 con 0x7fca780aef70 2024-03-29T11:24:15.666 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.682+0000 7fca767fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:15.666 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.682+0000 7fca767fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:15.667 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.682+0000 7fca767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:15.667 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.682+0000 7fca767fc640 10 monclient(hunting): my global_id is 4442 2024-03-29T11:24:15.667 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.682+0000 7fca767fc640 1 -- 172.21.15.17:0/10755211 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca7806d960 con 0x7fca780aef70 2024-03-29T11:24:15.667 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.682+0000 7fca767fc640 1 -- 172.21.15.17:0/10755211 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1947405419 0 0) 0x7fca780b0020 con 0x7fca780727a0 2024-03-29T11:24:15.667 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.682+0000 7fca767fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:15.667 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.682+0000 7fca767fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:15.667 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.682+0000 7fca767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:15.667 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.682+0000 7fca767fc640 10 monclient(hunting): my global_id is 4416 2024-03-29T11:24:15.667 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.682+0000 7fca767fc640 1 -- 172.21.15.17:0/10755211 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca780ad310 con 0x7fca780727a0 2024-03-29T11:24:15.667 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.682+0000 7fca767fc640 1 -- 172.21.15.17:0/10755211 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3348573333 0 0) 0x7fca60003520 con 0x7fca780ae490 2024-03-29T11:24:15.667 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.682+0000 7fca767fc640 1 -- 172.21.15.17:0/10755211 >> v1:172.21.15.17:6790/0 conn(0x7fca780aef70 legacy=0x7fca78072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:15.668 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.682+0000 7fca767fc640 1 -- 172.21.15.17:0/10755211 >> v1:172.21.15.17:6789/0 conn(0x7fca780727a0 legacy=0x7fca7814b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:15.668 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.682+0000 7fca767fc640 10 monclient: _finish_hunting 0 2024-03-29T11:24:15.668 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.682+0000 7fca767fc640 1 monclient: found mon.noname-c 2024-03-29T11:24:15.668 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.682+0000 7fca767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:15.668 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.682+0000 7fca767fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6789/0 2024-03-29T11:24:15.668 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.682+0000 7fca767fc640 1 -- 172.21.15.17:0/10755211 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca781432d0 con 0x7fca780ae490 2024-03-29T11:24:15.668 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.682+0000 7fca767fc640 10 monclient: _finish_auth 0 2024-03-29T11:24:15.668 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.682+0000 7fca767fc640 10 monclient: _check_auth_tickets 2024-03-29T11:24:15.668 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.682+0000 7fca767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:15.668 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.682+0000 7fca767fc640 1 -- 172.21.15.17:0/10755211 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fca68001610 con 0x7fca780ae490 2024-03-29T11:24:15.668 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.682+0000 7fca767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:15.668 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.682+0000 7fca767fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:15.669 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.682+0000 7fca767fc640 10 monclient: dump: 2024-03-29T11:24:15.669 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:24:15.669 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:15.669 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:15.669 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:15.669 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:15.669 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:15.669 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:24:15.669 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:24:15.669 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:24:15.669 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:15.670 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.682+0000 7fca7cab3640 5 monclient: authenticate success, global_id 4435 2024-03-29T11:24:15.670 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.682+0000 7fca7cab3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:15.670 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.682+0000 7fca767fc640 1 -- 172.21.15.17:0/10755211 <== mon.1 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca68001770 con 0x7fca780ae490 2024-03-29T11:24:15.670 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.682+0000 7fca767fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:15.670 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.682+0000 7fca7cab3640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:15.670 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.682+0000 7fca7cab3640 4 set_mon_vals no callback set 2024-03-29T11:24:15.670 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.683+0000 7fca767fc640 1 -- 172.21.15.17:0/10755211 <== mon.1 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fca68001bc0 con 0x7fca780ae490 2024-03-29T11:24:15.670 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.683+0000 7fca767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:15.670 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.683+0000 7fca767fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:15.670 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.683+0000 7fca767fc640 10 monclient: dump: 2024-03-29T11:24:15.670 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:24:15.670 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:15.670 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:15.671 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:15.671 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:15.671 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:15.671 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:24:15.671 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:24:15.671 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:24:15.671 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:15.671 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.683+0000 7fca7cab3640 10 monclient: shutdown 2024-03-29T11:24:15.671 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.683+0000 7fca7cab3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:15.671 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.683+0000 7fca7cab3640 1 -- 172.21.15.17:0/10755211 >> v1:172.21.15.94:6789/0 conn(0x7fca780ae490 legacy=0x7fca780713a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:15.671 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.683+0000 7fca7cab3640 1 -- 172.21.15.17:0/10755211 shutdown_connections 2024-03-29T11:24:15.671 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.683+0000 7fca7cab3640 1 -- 172.21.15.17:0/10755211 >> 172.21.15.17:0/10755211 conn(0x7fca780673e0 msgr2=0x7fca780b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:15.671 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.683+0000 7fca7cab3640 1 -- 172.21.15.17:0/10755211 shutdown_connections 2024-03-29T11:24:15.671 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.684+0000 7fca7cab3640 1 -- 172.21.15.17:0/10755211 wait complete. 2024-03-29T11:24:15.677 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.685+0000 7fca7cab3640 10 monclient: build_initial_monmap 2024-03-29T11:24:15.677 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.685+0000 7fca7cab3640 10 monclient: monmap: 2024-03-29T11:24:15.677 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:24:15.678 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:15.678 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:24:15.678 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:24:15.678 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:15.678 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:15.678 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:24:15.678 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:24:15.678 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:24:15.678 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:15.678 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.685+0000 7fca7cab3640 1 librados: starting msgr at 2024-03-29T11:24:15.678 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.685+0000 7fca7cab3640 1 librados: starting objecter 2024-03-29T11:24:15.678 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.685+0000 7fca7cab3640 1 Processor -- start 2024-03-29T11:24:15.678 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.686+0000 7fca7cab3640 1 -- start start 2024-03-29T11:24:15.679 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.686+0000 7fca7cab3640 1 librados: setting wanted keys 2024-03-29T11:24:15.679 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.686+0000 7fca7cab3640 1 librados: calling monclient init 2024-03-29T11:24:15.679 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.686+0000 7fca7cab3640 10 monclient: init 2024-03-29T11:24:15.679 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.687+0000 7fca7cab3640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:15.679 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.687+0000 7fca7cab3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:24:15.679 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.687+0000 7fca7cab3640 10 monclient(hunting): picked mon.noname-c con 0x7fca780727a0 addr v1:172.21.15.17:6790/0 2024-03-29T11:24:15.679 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.687+0000 7fca7cab3640 10 monclient(hunting): picked mon.noname-b con 0x7fca780aef70 addr v1:172.21.15.94:6789/0 2024-03-29T11:24:15.679 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.687+0000 7fca777fe640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7fca780727a0 0x7fca7813db50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:43630/0 (socket says 172.21.15.17:43630) 2024-03-29T11:24:15.679 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.687+0000 7fca76ffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fca780aef70 0x7fca7813c160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:43718/0 (socket says 172.21.15.17:43718) 2024-03-29T11:24:15.679 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.687+0000 7fca777fe640 1 -- 172.21.15.17:0/4021125920 learned_addr learned my addr 172.21.15.17:0/4021125920 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:24:15.679 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.687+0000 7fca7cab3640 10 monclient(hunting): picked mon.noname-a con 0x7fca781db750 addr v1:172.21.15.17:6789/0 2024-03-29T11:24:15.679 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.687+0000 7fca5ffff640 10 client.?.objecter ms_handle_connect 0x7fca780727a0 2024-03-29T11:24:15.679 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.687+0000 7fca5ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:15.680 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.687+0000 7fca5ffff640 10 client.?.objecter ms_handle_connect 0x7fca780aef70 2024-03-29T11:24:15.680 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.687+0000 7fca5ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:15.680 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.687+0000 7fca7cab3640 1 -- 172.21.15.17:0/4021125920 --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca781dbd90 con 0x7fca781db750 2024-03-29T11:24:15.680 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.687+0000 7fca7cab3640 1 -- 172.21.15.17:0/4021125920 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca7813cf80 con 0x7fca780aef70 2024-03-29T11:24:15.680 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.687+0000 7fca7cab3640 1 -- 172.21.15.17:0/4021125920 --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca7811c010 con 0x7fca780727a0 2024-03-29T11:24:15.680 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.687+0000 7fca7cab3640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:15.680 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.687+0000 7fca7cab3640 10 monclient(hunting): authenticate will time out at 1161.893735s 2024-03-29T11:24:15.680 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.688+0000 7fca5ffff640 10 client.?.objecter ms_handle_connect 0x7fca781db750 2024-03-29T11:24:15.680 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.688+0000 7fca5ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:15.680 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.688+0000 7fca5ffff640 1 -- 172.21.15.17:0/4021125920 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2534565296 0 0) 0x7fca7813cf80 con 0x7fca780aef70 2024-03-29T11:24:15.680 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.688+0000 7fca5ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:15.680 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.688+0000 7fca5ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:15.680 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.688+0000 7fca5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:15.681 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.688+0000 7fca5ffff640 10 monclient(hunting): my global_id is 4438 2024-03-29T11:24:15.681 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.688+0000 7fca5ffff640 1 -- 172.21.15.17:0/4021125920 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca64004eb0 con 0x7fca780aef70 2024-03-29T11:24:15.681 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.688+0000 7fca5ffff640 1 -- 172.21.15.17:0/4021125920 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 477108171 0 0) 0x7fca7811c010 con 0x7fca780727a0 2024-03-29T11:24:15.681 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.688+0000 7fca5ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:15.681 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.688+0000 7fca5ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:15.681 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.688+0000 7fca5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:15.681 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.688+0000 7fca5ffff640 10 monclient(hunting): my global_id is 4445 2024-03-29T11:24:15.681 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.688+0000 7fca5ffff640 1 -- 172.21.15.17:0/4021125920 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca7813cf80 con 0x7fca780727a0 2024-03-29T11:24:15.681 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.688+0000 7fca5ffff640 1 -- 172.21.15.17:0/4021125920 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 109745921 0 0) 0x7fca781dbd90 con 0x7fca781db750 2024-03-29T11:24:15.681 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.688+0000 7fca5ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:15.681 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.688+0000 7fca5ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:15.681 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.688+0000 7fca5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:15.681 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.688+0000 7fca5ffff640 10 monclient(hunting): my global_id is 4419 2024-03-29T11:24:15.681 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.688+0000 7fca5ffff640 1 -- 172.21.15.17:0/4021125920 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca7811c010 con 0x7fca781db750 2024-03-29T11:24:15.682 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.689+0000 7fca5ffff640 1 -- 172.21.15.17:0/4021125920 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2072415833 0 0) 0x7fca64004eb0 con 0x7fca780aef70 2024-03-29T11:24:15.682 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.689+0000 7fca5ffff640 1 -- 172.21.15.17:0/4021125920 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca781dbd90 con 0x7fca780aef70 2024-03-29T11:24:15.682 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.689+0000 7fca5ffff640 1 -- 172.21.15.17:0/4021125920 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fca70001b70 con 0x7fca780aef70 2024-03-29T11:24:15.682 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.689+0000 7fca5ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:15.682 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.689+0000 7fca5ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:15.682 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.689+0000 7fca5ffff640 10 monclient(hunting): dump: 2024-03-29T11:24:15.682 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:24:15.682 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:15.682 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:15.682 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:15.682 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:15.682 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:15.682 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:24:15.682 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:24:15.683 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:24:15.683 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:15.683 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.689+0000 7fca5ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:15.683 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.689+0000 7fca5ffff640 1 -- 172.21.15.17:0/4021125920 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1166595742 0 0) 0x7fca7813cf80 con 0x7fca780727a0 2024-03-29T11:24:15.683 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.689+0000 7fca5ffff640 1 -- 172.21.15.17:0/4021125920 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca64004eb0 con 0x7fca780727a0 2024-03-29T11:24:15.683 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.690+0000 7fca5ffff640 1 -- 172.21.15.17:0/4021125920 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fca68001a80 con 0x7fca780727a0 2024-03-29T11:24:15.683 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.690+0000 7fca5ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:15.683 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.690+0000 7fca5ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:15.683 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.690+0000 7fca5ffff640 10 monclient(hunting): dump: 2024-03-29T11:24:15.683 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:24:15.683 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:15.683 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:15.683 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:15.683 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:15.683 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:15.684 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:24:15.684 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:24:15.684 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:24:15.684 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:15.684 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.690+0000 7fca5ffff640 1 -- 172.21.15.17:0/4021125920 <== mon.1 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1736289036 0 0) 0x7fca781dbd90 con 0x7fca780aef70 2024-03-29T11:24:15.684 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.690+0000 7fca5ffff640 1 -- 172.21.15.17:0/4021125920 >> v1:172.21.15.17:6790/0 conn(0x7fca780727a0 legacy=0x7fca7813db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:15.684 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.690+0000 7fca5ffff640 1 -- 172.21.15.17:0/4021125920 >> v1:172.21.15.17:6789/0 conn(0x7fca781db750 legacy=0x7fca7813c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:15.684 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.690+0000 7fca5ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:24:15.684 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.690+0000 7fca5ffff640 1 monclient: found mon.b 2024-03-29T11:24:15.684 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.690+0000 7fca5ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:15.684 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.690+0000 7fca5ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:24:15.684 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.690+0000 7fca5ffff640 1 -- 172.21.15.17:0/4021125920 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca7811c2f0 con 0x7fca780aef70 2024-03-29T11:24:15.684 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.690+0000 7fca5ffff640 10 monclient: _finish_auth 0 2024-03-29T11:24:15.684 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.690+0000 7fca5ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:24:15.684 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.690+0000 7fca5ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:15.685 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.690+0000 7fca7cab3640 5 monclient: authenticate success, global_id 4438 2024-03-29T11:24:15.685 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.690+0000 7fca7cab3640 10 monclient: _renew_subs 2024-03-29T11:24:15.685 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.690+0000 7fca7cab3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:24:15.685 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.690+0000 7fca7cab3640 1 -- 172.21.15.17:0/4021125920 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca7813d200 con 0x7fca780aef70 2024-03-29T11:24:15.685 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.690+0000 7fca7cab3640 10 client.4438.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:15.685 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.690+0000 7fca7cab3640 10 monclient: _renew_subs 2024-03-29T11:24:15.685 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.690+0000 7fca7cab3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:24:15.685 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.690+0000 7fca7cab3640 1 -- 172.21.15.17:0/4021125920 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fca78070800 con 0x7fca780aef70 2024-03-29T11:24:15.685 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.690+0000 7fca7cab3640 1 librados: init done 2024-03-29T11:24:15.685 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.691+0000 7fca5ffff640 1 -- 172.21.15.17:0/4021125920 <== mon.1 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca70001cd0 con 0x7fca780aef70 2024-03-29T11:24:15.686 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.691+0000 7fca5ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:15.686 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.691+0000 7fca5ffff640 1 -- 172.21.15.17:0/4021125920 <== mon.1 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fca70004d00 con 0x7fca780aef70 2024-03-29T11:24:15.686 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.691+0000 7fca5ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:15.686 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.691+0000 7fca5ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:15.686 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.691+0000 7fca5ffff640 10 monclient: dump: 2024-03-29T11:24:15.686 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:24:15.686 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:15.686 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:15.686 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:15.686 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:15.686 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:15.686 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:24:15.686 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:24:15.686 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:24:15.687 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:15.687 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.691+0000 7fca75ffb640 4 set_mon_vals no callback set 2024-03-29T11:24:15.687 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.692+0000 7fca7cab3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:15.687 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.692+0000 7fca7cab3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:15.687 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.692+0000 7fca7cab3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:24:15.687 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.692+0000 7fca7cab3640 1 -- 172.21.15.17:0/4021125920 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca58005080 con 0x7fca780aef70 2024-03-29T11:24:15.687 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.694+0000 7fca5ffff640 1 -- 172.21.15.17:0/4021125920 <== mon.1 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7fca7001acd0 con 0x7fca780aef70 2024-03-29T11:24:15.687 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.695+0000 7fca5ffff640 1 -- 172.21.15.17:0/4021125920 <== mon.1 v1:172.21.15.94:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7fca70084bf0 con 0x7fca780aef70 2024-03-29T11:24:15.687 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.695+0000 7fca5ffff640 10 client.4438.objecter ms_dispatch 0x7fca78000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:15.687 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.695+0000 7fca5ffff640 3 client.4438.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:15.687 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.695+0000 7fca5ffff640 3 client.4438.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:15.687 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.695+0000 7fca5ffff640 20 client.4438.objecter dump_active .. 0 homeless 2024-03-29T11:24:15.688 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.695+0000 7fca777fe640 10 monclient: get_auth_request con 0x7fca6406de30 auth_method 0 2024-03-29T11:24:15.688 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.696+0000 7fca5ffff640 10 client.4438.objecter ms_handle_connect 0x7fca6406de30 2024-03-29T11:24:15.688 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.702+0000 7fca5ffff640 1 -- 172.21.15.17:0/4021125920 <== mon.1 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fca70050530 con 0x7fca780aef70 2024-03-29T11:24:15.688 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.702+0000 7fca5ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:15.689 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:15.702+0000 7fca5ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:16.095 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:16.113+0000 7fca7cab3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:16.095 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:16.113+0000 7fca7cab3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:16.095 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:16.113+0000 7fca7cab3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:24:16.095 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:16.113+0000 7fca7cab3640 1 -- 172.21.15.17:0/4021125920 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fca58005330 con 0x7fca780aef70 2024-03-29T11:24:16.096 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:24:16.097 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:16.097 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:16.114+0000 7fca5ffff640 1 -- 172.21.15.17:0/4021125920 <== mon.1 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7fca700842a0 con 0x7fca780aef70 2024-03-29T11:24:16.097 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:16.114+0000 7fca5ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:16.097 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:16.114+0000 7fca5ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:16.100 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:16.118+0000 7fca7cab3640 10 librados: watch_flush enter 2024-03-29T11:24:16.100 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:16.118+0000 7fca7cab3640 10 librados: watch_flush exit 2024-03-29T11:24:16.100 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:16.118+0000 7fca7cab3640 20 client.4438.objecter shutdown clearing up homeless session... 2024-03-29T11:24:16.100 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:16.118+0000 7fca7cab3640 10 client.4438.objecter successfully canceled tick 2024-03-29T11:24:16.101 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:16.118+0000 7fca7cab3640 1 -- 172.21.15.17:0/4021125920 >> v1:172.21.15.17:6816/3236185052 conn(0x7fca6406de30 legacy=0x7fca640702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:16.101 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:16.118+0000 7fca7cab3640 10 monclient: shutdown 2024-03-29T11:24:16.101 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:16.118+0000 7fca7cab3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:16.101 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:16.118+0000 7fca7cab3640 1 -- 172.21.15.17:0/4021125920 >> v1:172.21.15.94:6789/0 conn(0x7fca780aef70 legacy=0x7fca7813c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:16.101 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:16.118+0000 7fca7cab3640 1 -- 172.21.15.17:0/4021125920 shutdown_connections 2024-03-29T11:24:16.101 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:16.119+0000 7fca7cab3640 1 -- 172.21.15.17:0/4021125920 >> 172.21.15.17:0/4021125920 conn(0x7fca780673e0 msgr2=0x7fca780b2c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:16.101 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:16.119+0000 7fca7cab3640 1 -- 172.21.15.17:0/4021125920 shutdown_connections 2024-03-29T11:24:16.102 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:16.119+0000 7fca7cab3640 1 -- 172.21.15.17:0/4021125920 wait complete. 2024-03-29T11:24:16.102 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:16.119+0000 7fca7cab3640 1 librados: shutdown 2024-03-29T11:24:16.119 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:22.927 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:23.076 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.094+0000 7f5082c61640 10 monclient: get_monmap_and_config 2024-03-29T11:24:23.076 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.094+0000 7f5082c61640 10 monclient: build_initial_monmap 2024-03-29T11:24:23.076 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.094+0000 7f5082c61640 10 monclient: monmap: 2024-03-29T11:24:23.076 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:24:23.077 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:23.077 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:24:23.095817+0000 2024-03-29T11:24:23.077 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:24:23.095817+0000 2024-03-29T11:24:23.077 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:23.077 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:23.077 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:24:23.077 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:24:23.077 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:24:23.077 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:23.078 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.095+0000 7f5082c61640 1 Processor -- start 2024-03-29T11:24:23.078 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.096+0000 7f5082c61640 1 -- start start 2024-03-29T11:24:23.078 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.096+0000 7f5082c61640 10 monclient: init 2024-03-29T11:24:23.078 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.096+0000 7f5082c61640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:23.078 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.096+0000 7f5082c61640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:24:23.078 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.097+0000 7f5082c61640 10 monclient(hunting): picked mon.noname-c con 0x7f50740389e0 addr v1:172.21.15.94:6789/0 2024-03-29T11:24:23.083 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.098+0000 7f5082c61640 10 monclient(hunting): picked mon.noname-a con 0x7f5074038dc0 addr v1:172.21.15.17:6789/0 2024-03-29T11:24:23.083 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.098+0000 7f5082c61640 10 monclient(hunting): picked mon.noname-b con 0x7f5074030660 addr v1:172.21.15.17:6790/0 2024-03-29T11:24:23.083 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.098+0000 7f5082c61640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50740c54c0 con 0x7f5074038dc0 2024-03-29T11:24:23.083 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.098+0000 7f5082c61640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5074036430 con 0x7f50740389e0 2024-03-29T11:24:23.084 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.098+0000 7f5082c61640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5074039840 con 0x7f5074030660 2024-03-29T11:24:23.084 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.098+0000 7f5082c61640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:23.084 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.098+0000 7f5082c61640 10 monclient(hunting): authenticate will time out at 1169.303448s 2024-03-29T11:24:23.084 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.101+0000 7f5081c5f640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f50740389e0 0x7f5074033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:45839/0 (socket says 172.21.15.17:45839) 2024-03-29T11:24:23.084 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.101+0000 7f5081c5f640 1 -- 172.21.15.17:0/811043223 learned_addr learned my addr 172.21.15.17:0/811043223 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:24:23.084 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.102+0000 7f5080c5d640 1 -- 172.21.15.17:0/811043223 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2434500074 0 0) 0x7f5074036430 con 0x7f50740389e0 2024-03-29T11:24:23.084 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.102+0000 7f5080c5d640 10 monclient(hunting): _negotiate 2024-03-29T11:24:23.084 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.102+0000 7f5080c5d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:23.084 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.102+0000 7f5080c5d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:23.084 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.102+0000 7f5080c5d640 10 monclient(hunting): my global_id is 4441 2024-03-29T11:24:23.084 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.102+0000 7f5080c5d640 1 -- 172.21.15.17:0/811043223 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5068003520 con 0x7f50740389e0 2024-03-29T11:24:23.088 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.102+0000 7f5080c5d640 1 -- 172.21.15.17:0/811043223 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3463806577 0 0) 0x7f5068003520 con 0x7f50740389e0 2024-03-29T11:24:23.088 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.102+0000 7f5080c5d640 1 -- 172.21.15.17:0/811043223 >> v1:172.21.15.17:6790/0 conn(0x7f5074030660 legacy=0x7f50740d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.088 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.102+0000 7f5080c5d640 1 -- 172.21.15.17:0/811043223 >> v1:172.21.15.17:6789/0 conn(0x7f5074038dc0 legacy=0x7f50740cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.088 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.103+0000 7f5080c5d640 10 monclient: _finish_hunting 0 2024-03-29T11:24:23.088 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.103+0000 7f5080c5d640 1 monclient: found mon.noname-c 2024-03-29T11:24:23.088 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.103+0000 7f5080c5d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:23.088 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.103+0000 7f5080c5d640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6789/0 2024-03-29T11:24:23.088 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.103+0000 7f5080c5d640 1 -- 172.21.15.17:0/811043223 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5074035690 con 0x7f50740389e0 2024-03-29T11:24:23.088 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.103+0000 7f5080c5d640 10 monclient: _finish_auth 0 2024-03-29T11:24:23.088 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.103+0000 7f5080c5d640 10 monclient: _check_auth_tickets 2024-03-29T11:24:23.088 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.103+0000 7f5080c5d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:23.088 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.103+0000 7f5080c5d640 1 -- 172.21.15.17:0/811043223 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f5078001690 con 0x7f50740389e0 2024-03-29T11:24:23.088 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.103+0000 7f5080c5d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.089 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.103+0000 7f5080c5d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:23.089 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.103+0000 7f5080c5d640 10 monclient: dump: 2024-03-29T11:24:23.089 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:24:23.089 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:23.089 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:23.089 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:23.089 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.089 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:23.089 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:24:23.089 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:24:23.089 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:24:23.089 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:23.089 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.103+0000 7f5080c5d640 1 -- 172.21.15.17:0/811043223 <== mon.1 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5078001920 con 0x7f50740389e0 2024-03-29T11:24:23.089 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.103+0000 7f5080c5d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:23.089 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.103+0000 7f5080c5d640 1 -- 172.21.15.17:0/811043223 <== mon.1 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f5078001cb0 con 0x7f50740389e0 2024-03-29T11:24:23.090 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.103+0000 7f5080c5d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.090 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.103+0000 7f5080c5d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:23.090 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.103+0000 7f5080c5d640 10 monclient: dump: 2024-03-29T11:24:23.090 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:24:23.090 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:23.090 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:23.090 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:23.090 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.090 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:23.090 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:24:23.090 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:24:23.090 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:24:23.090 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:23.090 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.104+0000 7f5082c61640 5 monclient: authenticate success, global_id 4441 2024-03-29T11:24:23.091 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.104+0000 7f5082c61640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:23.091 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.104+0000 7f5082c61640 4 set_mon_vals no callback set 2024-03-29T11:24:23.091 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.104+0000 7f5082c61640 10 monclient: shutdown 2024-03-29T11:24:23.091 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.104+0000 7f5082c61640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:23.091 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.104+0000 7f5082c61640 1 -- 172.21.15.17:0/811043223 >> v1:172.21.15.94:6789/0 conn(0x7f50740389e0 legacy=0x7f5074033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.091 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.105+0000 7f5082c61640 1 -- 172.21.15.17:0/811043223 shutdown_connections 2024-03-29T11:24:23.091 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.105+0000 7f5082c61640 1 -- 172.21.15.17:0/811043223 >> 172.21.15.17:0/811043223 conn(0x7f5074005c00 msgr2=0x7f5074006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:23.091 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.105+0000 7f5082c61640 1 -- 172.21.15.17:0/811043223 shutdown_connections 2024-03-29T11:24:23.091 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.105+0000 7f5082c61640 1 -- 172.21.15.17:0/811043223 wait complete. 2024-03-29T11:24:23.091 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.106+0000 7f5082c61640 10 monclient: build_initial_monmap 2024-03-29T11:24:23.091 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.106+0000 7f5082c61640 10 monclient: monmap: 2024-03-29T11:24:23.091 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:24:23.091 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:23.091 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:24:23.091 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:24:23.092 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:23.092 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:23.092 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:24:23.092 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:24:23.092 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:24:23.092 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:23.092 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.106+0000 7f5082c61640 1 librados: starting msgr at 2024-03-29T11:24:23.092 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.106+0000 7f5082c61640 1 librados: starting objecter 2024-03-29T11:24:23.092 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.106+0000 7f5082c61640 1 Processor -- start 2024-03-29T11:24:23.093 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.107+0000 7f5082c61640 1 -- start start 2024-03-29T11:24:23.093 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.107+0000 7f5082c61640 1 librados: setting wanted keys 2024-03-29T11:24:23.093 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.107+0000 7f5082c61640 1 librados: calling monclient init 2024-03-29T11:24:23.093 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.107+0000 7f5082c61640 10 monclient: init 2024-03-29T11:24:23.093 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.107+0000 7f5082c61640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:23.093 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.107+0000 7f5082c61640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:24:23.093 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.107+0000 7f5082c61640 10 monclient(hunting): picked mon.noname-c con 0x7f5074030660 addr v1:172.21.15.17:6790/0 2024-03-29T11:24:23.093 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.107+0000 7f5082c61640 10 monclient(hunting): picked mon.noname-b con 0x7f50740389e0 addr v1:172.21.15.94:6789/0 2024-03-29T11:24:23.093 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.107+0000 7f5082c61640 10 monclient(hunting): picked mon.noname-a con 0x7f507415d530 addr v1:172.21.15.17:6789/0 2024-03-29T11:24:23.093 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.107+0000 7f5082c61640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50740a6160 con 0x7f507415d530 2024-03-29T11:24:23.093 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.107+0000 7f5082c61640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50740c3d20 con 0x7f50740389e0 2024-03-29T11:24:23.093 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.107+0000 7f5082c61640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50740a63d0 con 0x7f5074030660 2024-03-29T11:24:23.093 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.107+0000 7f5082c61640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:23.093 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.107+0000 7f5082c61640 10 monclient(hunting): authenticate will time out at 1169.312768s 2024-03-29T11:24:23.096 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.114+0000 7f5082460640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f507415d530 0x7f50740c3610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:44019/0 (socket says 172.21.15.17:44019) 2024-03-29T11:24:23.097 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.114+0000 7f5082460640 1 -- 172.21.15.17:0/2305125834 learned_addr learned my addr 172.21.15.17:0/2305125834 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:24:23.097 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.116+0000 7f50727fc640 10 client.?.objecter ms_handle_connect 0x7f507415d530 2024-03-29T11:24:23.097 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.116+0000 7f50727fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:23.099 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.117+0000 7f50727fc640 10 client.?.objecter ms_handle_connect 0x7f5074030660 2024-03-29T11:24:23.099 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.118+0000 7f50727fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:23.099 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.118+0000 7f50727fc640 1 -- 172.21.15.17:0/2305125834 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1588043258 0 0) 0x7f50740a63d0 con 0x7f5074030660 2024-03-29T11:24:23.099 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.118+0000 7f50727fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:23.099 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.118+0000 7f50727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:23.099 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.118+0000 7f50727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:23.099 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.118+0000 7f50727fc640 10 monclient(hunting): my global_id is 4451 2024-03-29T11:24:23.100 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.118+0000 7f50727fc640 1 -- 172.21.15.17:0/2305125834 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5060004eb0 con 0x7f5074030660 2024-03-29T11:24:23.101 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.118+0000 7f50727fc640 1 -- 172.21.15.17:0/2305125834 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2612187014 0 0) 0x7f50740a6160 con 0x7f507415d530 2024-03-29T11:24:23.101 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.118+0000 7f50727fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:23.101 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.118+0000 7f50727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:23.101 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.118+0000 7f50727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:23.101 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.118+0000 7f50727fc640 10 monclient(hunting): my global_id is 4422 2024-03-29T11:24:23.101 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.118+0000 7f50727fc640 1 -- 172.21.15.17:0/2305125834 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50740a63d0 con 0x7f507415d530 2024-03-29T11:24:23.101 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.118+0000 7f50727fc640 1 -- 172.21.15.17:0/2305125834 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 466053981 0 0) 0x7f5060004eb0 con 0x7f5074030660 2024-03-29T11:24:23.101 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.118+0000 7f50727fc640 1 -- 172.21.15.17:0/2305125834 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50740a6160 con 0x7f5074030660 2024-03-29T11:24:23.101 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.118+0000 7f50727fc640 10 client.?.objecter ms_handle_connect 0x7f50740389e0 2024-03-29T11:24:23.101 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.118+0000 7f50727fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:23.101 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.118+0000 7f50727fc640 1 -- 172.21.15.17:0/2305125834 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f5078001550 con 0x7f5074030660 2024-03-29T11:24:23.101 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.118+0000 7f50727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.101 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.118+0000 7f50727fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:23.102 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.118+0000 7f50727fc640 10 monclient(hunting): dump: 2024-03-29T11:24:23.102 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:24:23.102 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:23.102 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:23.102 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:23.102 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.102 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:23.102 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:24:23.102 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:24:23.102 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:24:23.102 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:23.102 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.118+0000 7f50727fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:23.102 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.119+0000 7f50727fc640 1 -- 172.21.15.17:0/2305125834 <== mon.2 v1:172.21.15.17:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 52154892 0 0) 0x7f50740a6160 con 0x7f5074030660 2024-03-29T11:24:23.103 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.119+0000 7f50727fc640 1 -- 172.21.15.17:0/2305125834 >> v1:172.21.15.94:6789/0 conn(0x7f50740389e0 legacy=0x7f50740c2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.103 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.119+0000 7f50727fc640 1 -- 172.21.15.17:0/2305125834 >> v1:172.21.15.17:6789/0 conn(0x7f507415d530 legacy=0x7f50740c3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.103 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.119+0000 7f50727fc640 10 monclient: _finish_hunting 0 2024-03-29T11:24:23.103 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.119+0000 7f50727fc640 1 monclient: found mon.c 2024-03-29T11:24:23.103 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.119+0000 7f50727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:23.103 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.119+0000 7f50727fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:24:23.103 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.119+0000 7f50727fc640 1 -- 172.21.15.17:0/2305125834 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50740a66b0 con 0x7f5074030660 2024-03-29T11:24:23.103 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.119+0000 7f50727fc640 10 monclient: _finish_auth 0 2024-03-29T11:24:23.103 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.119+0000 7f50727fc640 10 monclient: _check_auth_tickets 2024-03-29T11:24:23.103 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.119+0000 7f50727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:23.105 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.120+0000 7f5082c61640 5 monclient: authenticate success, global_id 4451 2024-03-29T11:24:23.105 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.120+0000 7f5082c61640 10 monclient: _renew_subs 2024-03-29T11:24:23.105 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.120+0000 7f5082c61640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:24:23.105 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.120+0000 7f5082c61640 1 -- 172.21.15.17:0/2305125834 --> v1:172.21.15.17:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50740c7930 con 0x7f5074030660 2024-03-29T11:24:23.105 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.120+0000 7f50727fc640 1 -- 172.21.15.17:0/2305125834 <== mon.2 v1:172.21.15.17:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50780016b0 con 0x7f5074030660 2024-03-29T11:24:23.106 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.120+0000 7f50727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:23.106 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.120+0000 7f5082c61640 10 client.4451.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:23.106 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.120+0000 7f5073fff640 4 set_mon_vals no callback set 2024-03-29T11:24:23.106 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.120+0000 7f5082c61640 10 monclient: _renew_subs 2024-03-29T11:24:23.106 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.120+0000 7f5082c61640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:24:23.106 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.120+0000 7f5082c61640 1 -- 172.21.15.17:0/2305125834 --> v1:172.21.15.17:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f50740ca680 con 0x7f5074030660 2024-03-29T11:24:23.106 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.120+0000 7f5082c61640 1 librados: init done 2024-03-29T11:24:23.106 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.121+0000 7f50727fc640 1 -- 172.21.15.17:0/2305125834 <== mon.2 v1:172.21.15.17:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f5078004d90 con 0x7f5074030660 2024-03-29T11:24:23.106 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.121+0000 7f50727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.106 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.121+0000 7f50727fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:23.106 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.121+0000 7f50727fc640 10 monclient: dump: 2024-03-29T11:24:23.106 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:24:23.106 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:23.106 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:23.107 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:23.107 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.107 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:23.107 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:24:23.107 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:24:23.107 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:24:23.107 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:23.107 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.122+0000 7f50727fc640 1 -- 172.21.15.17:0/2305125834 <== mon.2 v1:172.21.15.17:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f507801ad60 con 0x7f5074030660 2024-03-29T11:24:23.107 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.123+0000 7f508145e640 10 monclient: get_auth_request con 0x7f506006d8a0 auth_method 0 2024-03-29T11:24:23.107 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.123+0000 7f5082c61640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:23.107 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.123+0000 7f5082c61640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:23.107 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.123+0000 7f5082c61640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:24:23.107 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.123+0000 7f5082c61640 1 -- 172.21.15.17:0/2305125834 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5074007b90 con 0x7f5074030660 2024-03-29T11:24:23.107 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.123+0000 7f50727fc640 10 client.4451.objecter ms_handle_connect 0x7f506006d8a0 2024-03-29T11:24:23.107 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.123+0000 7f50727fc640 1 -- 172.21.15.17:0/2305125834 <== mon.2 v1:172.21.15.17:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f5078085b70 con 0x7f5074030660 2024-03-29T11:24:23.108 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.123+0000 7f50727fc640 10 client.4451.objecter ms_dispatch 0x7f507c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:23.108 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.123+0000 7f50727fc640 3 client.4451.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:23.108 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.123+0000 7f50727fc640 3 client.4451.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:23.108 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.123+0000 7f50727fc640 20 client.4451.objecter dump_active .. 0 homeless 2024-03-29T11:24:23.113 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.131+0000 7f50727fc640 1 -- 172.21.15.17:0/2305125834 <== mon.2 v1:172.21.15.17:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5078050420 con 0x7f5074030660 2024-03-29T11:24:23.113 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.132+0000 7f50727fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:23.113 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.132+0000 7f50727fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:23.495 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.513+0000 7f5082c61640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:23.495 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.513+0000 7f5082c61640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:23.495 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.513+0000 7f5082c61640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:24:23.495 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.513+0000 7f5082c61640 1 -- 172.21.15.17:0/2305125834 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5074005120 con 0x7f5074030660 2024-03-29T11:24:23.496 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.514+0000 7f50727fc640 1 -- 172.21.15.17:0/2305125834 <== mon.2 v1:172.21.15.17:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f5078052350 con 0x7f5074030660 2024-03-29T11:24:23.496 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.514+0000 7f50727fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:23.496 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.515+0000 7f50727fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:23.496 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:24:23.496 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:23.499 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.518+0000 7f5082c61640 10 librados: watch_flush enter 2024-03-29T11:24:23.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.518+0000 7f5082c61640 10 librados: watch_flush exit 2024-03-29T11:24:23.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.518+0000 7f5082c61640 20 client.4451.objecter shutdown clearing up homeless session... 2024-03-29T11:24:23.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.518+0000 7f5082c61640 10 client.4451.objecter successfully canceled tick 2024-03-29T11:24:23.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.518+0000 7f5082c61640 1 -- 172.21.15.17:0/2305125834 >> v1:172.21.15.17:6816/3236185052 conn(0x7f506006d8a0 legacy=0x7f506006fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.501 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.518+0000 7f5082c61640 10 monclient: shutdown 2024-03-29T11:24:23.501 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.518+0000 7f5082c61640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:23.501 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.518+0000 7f5082c61640 1 -- 172.21.15.17:0/2305125834 >> v1:172.21.15.17:6790/0 conn(0x7f5074030660 legacy=0x7f507415ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.501 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.519+0000 7f5082c61640 1 -- 172.21.15.17:0/2305125834 shutdown_connections 2024-03-29T11:24:23.501 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.519+0000 7f5082c61640 1 -- 172.21.15.17:0/2305125834 >> 172.21.15.17:0/2305125834 conn(0x7f5074005c00 msgr2=0x7f507402f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:23.501 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.519+0000 7f5082c61640 1 -- 172.21.15.17:0/2305125834 shutdown_connections 2024-03-29T11:24:23.501 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.519+0000 7f5082c61640 1 -- 172.21.15.17:0/2305125834 wait complete. 2024-03-29T11:24:23.501 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:23.519+0000 7f5082c61640 1 librados: shutdown 2024-03-29T11:24:23.515 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:30.323 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:30.457 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.475+0000 7fa544965640 10 monclient: get_monmap_and_config 2024-03-29T11:24:30.458 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.475+0000 7fa544965640 10 monclient: build_initial_monmap 2024-03-29T11:24:30.458 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.475+0000 7fa544965640 10 monclient: monmap: 2024-03-29T11:24:30.458 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:24:30.458 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:30.458 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:24:30.476759+0000 2024-03-29T11:24:30.458 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:24:30.476759+0000 2024-03-29T11:24:30.458 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:30.458 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:30.458 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:24:30.458 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:24:30.458 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:24:30.458 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:30.459 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.476+0000 7fa544965640 1 Processor -- start 2024-03-29T11:24:30.459 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.476+0000 7fa544965640 1 -- start start 2024-03-29T11:24:30.459 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.476+0000 7fa544965640 10 monclient: init 2024-03-29T11:24:30.459 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.477+0000 7fa544965640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:30.459 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.477+0000 7fa544965640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:24:30.460 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.477+0000 7fa544965640 10 monclient(hunting): picked mon.noname-c con 0x7fa5400a9750 addr v1:172.21.15.94:6789/0 2024-03-29T11:24:30.460 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.477+0000 7fa53e59c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fa5400a9750 0x7fa54013dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:49446/0 (socket says 172.21.15.17:49446) 2024-03-29T11:24:30.460 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.478+0000 7fa544965640 10 monclient(hunting): picked mon.noname-a con 0x7fa54013e1e0 addr v1:172.21.15.17:6789/0 2024-03-29T11:24:30.460 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.478+0000 7fa544965640 10 monclient(hunting): picked mon.noname-b con 0x7fa540071c30 addr v1:172.21.15.17:6790/0 2024-03-29T11:24:30.460 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.478+0000 7fa53e59c640 1 -- 172.21.15.17:0/748811886 learned_addr learned my addr 172.21.15.17:0/748811886 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:24:30.461 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.478+0000 7fa544965640 1 -- 172.21.15.17:0/748811886 --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5400ab300 con 0x7fa54013e1e0 2024-03-29T11:24:30.461 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.478+0000 7fa544965640 1 -- 172.21.15.17:0/748811886 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5400ac2f0 con 0x7fa5400a9750 2024-03-29T11:24:30.461 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.478+0000 7fa544965640 1 -- 172.21.15.17:0/748811886 --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5400ad8d0 con 0x7fa540071c30 2024-03-29T11:24:30.461 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.478+0000 7fa544965640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:30.461 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.478+0000 7fa544965640 10 monclient(hunting): authenticate will time out at 1176.683589s 2024-03-29T11:24:30.461 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.479+0000 7fa53dd9b640 1 -- 172.21.15.17:0/748811886 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3988052894 0 0) 0x7fa5400ad8d0 con 0x7fa540071c30 2024-03-29T11:24:30.461 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.479+0000 7fa53dd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:24:30.461 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.479+0000 7fa53dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:30.461 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.479+0000 7fa53dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:30.461 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.479+0000 7fa53dd9b640 10 monclient(hunting): my global_id is 4454 2024-03-29T11:24:30.461 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.479+0000 7fa53dd9b640 1 -- 172.21.15.17:0/748811886 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa52c003440 con 0x7fa540071c30 2024-03-29T11:24:30.461 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.479+0000 7fa53dd9b640 1 -- 172.21.15.17:0/748811886 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 326232555 0 0) 0x7fa5400ac2f0 con 0x7fa5400a9750 2024-03-29T11:24:30.461 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.479+0000 7fa53dd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:24:30.461 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.479+0000 7fa53dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:30.462 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.479+0000 7fa53dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:30.462 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.479+0000 7fa53dd9b640 10 monclient(hunting): my global_id is 4447 2024-03-29T11:24:30.462 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.479+0000 7fa53dd9b640 1 -- 172.21.15.17:0/748811886 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5400ad8d0 con 0x7fa5400a9750 2024-03-29T11:24:30.462 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.479+0000 7fa53dd9b640 1 -- 172.21.15.17:0/748811886 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2641106973 0 0) 0x7fa5400ad8d0 con 0x7fa5400a9750 2024-03-29T11:24:30.462 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.479+0000 7fa53dd9b640 1 -- 172.21.15.17:0/748811886 >> v1:172.21.15.17:6790/0 conn(0x7fa540071c30 legacy=0x7fa5400ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:30.462 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.479+0000 7fa53dd9b640 1 -- 172.21.15.17:0/748811886 >> v1:172.21.15.17:6789/0 conn(0x7fa54013e1e0 legacy=0x7fa5400ac710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:24:30.463 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.481+0000 7fa53dd9b640 10 monclient: _finish_hunting 0 2024-03-29T11:24:30.463 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.481+0000 7fa53dd9b640 1 monclient: found mon.noname-c 2024-03-29T11:24:30.463 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.481+0000 7fa53dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:30.463 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.481+0000 7fa53dd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6789/0 2024-03-29T11:24:30.463 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.481+0000 7fa53dd9b640 1 -- 172.21.15.17:0/748811886 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5400ada60 con 0x7fa5400a9750 2024-03-29T11:24:30.464 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.481+0000 7fa53dd9b640 10 monclient: _finish_auth 0 2024-03-29T11:24:30.464 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.481+0000 7fa53dd9b640 10 monclient: _check_auth_tickets 2024-03-29T11:24:30.464 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.481+0000 7fa53dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:30.464 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.481+0000 7fa53dd9b640 1 -- 172.21.15.17:0/748811886 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fa5300015a0 con 0x7fa5400a9750 2024-03-29T11:24:30.464 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.481+0000 7fa53dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:30.464 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.481+0000 7fa53dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:30.464 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.481+0000 7fa53dd9b640 10 monclient: dump: 2024-03-29T11:24:30.464 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:24:30.464 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:30.464 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:30.464 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:30.464 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:30.464 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:30.464 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:24:30.464 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:24:30.464 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:24:30.465 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:30.465 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.481+0000 7fa544965640 5 monclient: authenticate success, global_id 4447 2024-03-29T11:24:30.465 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.481+0000 7fa544965640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:30.465 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.482+0000 7fa53dd9b640 1 -- 172.21.15.17:0/748811886 <== mon.1 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa530001700 con 0x7fa5400a9750 2024-03-29T11:24:30.466 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.482+0000 7fa53dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:30.466 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.482+0000 7fa53dd9b640 1 -- 172.21.15.17:0/748811886 <== mon.1 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fa530001b10 con 0x7fa5400a9750 2024-03-29T11:24:30.466 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.482+0000 7fa53dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:30.466 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.482+0000 7fa53dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:30.466 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.482+0000 7fa53dd9b640 10 monclient: dump: 2024-03-29T11:24:30.466 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:24:30.466 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:30.466 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:30.466 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:30.466 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:30.466 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:30.466 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:24:30.466 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:24:30.466 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:24:30.466 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:30.467 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.482+0000 7fa544965640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:30.467 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.482+0000 7fa544965640 4 set_mon_vals no callback set 2024-03-29T11:24:30.467 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.483+0000 7fa544965640 10 monclient: shutdown 2024-03-29T11:24:30.467 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.483+0000 7fa544965640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:30.467 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.483+0000 7fa544965640 1 -- 172.21.15.17:0/748811886 >> v1:172.21.15.94:6789/0 conn(0x7fa5400a9750 legacy=0x7fa54013dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:30.467 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.483+0000 7fa544965640 1 -- 172.21.15.17:0/748811886 shutdown_connections 2024-03-29T11:24:30.467 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.483+0000 7fa544965640 1 -- 172.21.15.17:0/748811886 >> 172.21.15.17:0/748811886 conn(0x7fa54006b660 msgr2=0x7fa54006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:30.467 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.483+0000 7fa544965640 1 -- 172.21.15.17:0/748811886 shutdown_connections 2024-03-29T11:24:30.467 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.484+0000 7fa544965640 1 -- 172.21.15.17:0/748811886 wait complete. 2024-03-29T11:24:30.468 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.484+0000 7fa544965640 10 monclient: build_initial_monmap 2024-03-29T11:24:30.468 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.484+0000 7fa544965640 10 monclient: monmap: 2024-03-29T11:24:30.468 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:24:30.468 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:30.468 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:24:30.468 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:24:30.468 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:30.468 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:30.468 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:24:30.468 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:24:30.468 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:24:30.468 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:30.469 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.485+0000 7fa544965640 1 librados: starting msgr at 2024-03-29T11:24:30.469 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.485+0000 7fa544965640 1 librados: starting objecter 2024-03-29T11:24:30.469 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.485+0000 7fa544965640 1 Processor -- start 2024-03-29T11:24:30.471 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.485+0000 7fa544965640 1 -- start start 2024-03-29T11:24:30.471 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.485+0000 7fa544965640 1 librados: setting wanted keys 2024-03-29T11:24:30.471 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.485+0000 7fa544965640 1 librados: calling monclient init 2024-03-29T11:24:30.471 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.485+0000 7fa544965640 10 monclient: init 2024-03-29T11:24:30.471 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.486+0000 7fa544965640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:30.471 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.486+0000 7fa544965640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:30.471 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.487+0000 7fa544965640 10 monclient(hunting): picked mon.noname-a con 0x7fa540071c30 addr v1:172.21.15.17:6789/0 2024-03-29T11:24:30.471 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.487+0000 7fa544965640 10 monclient(hunting): picked mon.noname-b con 0x7fa5400a9750 addr v1:172.21.15.94:6789/0 2024-03-29T11:24:30.471 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.487+0000 7fa544965640 10 monclient(hunting): picked mon.noname-c con 0x7fa54013e1e0 addr v1:172.21.15.17:6790/0 2024-03-29T11:24:30.472 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.487+0000 7fa544965640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa540134fc0 con 0x7fa540071c30 2024-03-29T11:24:30.472 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.487+0000 7fa544965640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa540135210 con 0x7fa5400a9750 2024-03-29T11:24:30.472 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.487+0000 7fa544965640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5401354f0 con 0x7fa54013e1e0 2024-03-29T11:24:30.472 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.487+0000 7fa544965640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:30.472 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.487+0000 7fa544965640 10 monclient(hunting): authenticate will time out at 1176.691983s 2024-03-29T11:24:30.472 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.487+0000 7fa537fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fa5400a9750 0x7fa540136e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:50298/0 (socket says 172.21.15.17:50298) 2024-03-29T11:24:30.472 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.487+0000 7fa537fff640 1 -- 172.21.15.17:0/3449876402 learned_addr learned my addr 172.21.15.17:0/3449876402 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:24:30.472 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.487+0000 7fa536ffd640 10 client.?.objecter ms_handle_connect 0x7fa5400a9750 2024-03-29T11:24:30.472 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.487+0000 7fa536ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:30.472 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.487+0000 7fa536ffd640 10 client.?.objecter ms_handle_connect 0x7fa54013e1e0 2024-03-29T11:24:30.472 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.487+0000 7fa536ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:30.472 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.487+0000 7fa536ffd640 1 -- 172.21.15.17:0/3449876402 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 775280214 0 0) 0x7fa540135210 con 0x7fa5400a9750 2024-03-29T11:24:30.472 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.487+0000 7fa536ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:30.472 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.487+0000 7fa536ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:30.472 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.487+0000 7fa536ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:30.473 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.487+0000 7fa536ffd640 10 monclient(hunting): my global_id is 4450 2024-03-29T11:24:30.473 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.487+0000 7fa536ffd640 1 -- 172.21.15.17:0/3449876402 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa520004eb0 con 0x7fa5400a9750 2024-03-29T11:24:30.473 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.488+0000 7fa536ffd640 1 -- 172.21.15.17:0/3449876402 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1319226235 0 0) 0x7fa5401354f0 con 0x7fa54013e1e0 2024-03-29T11:24:30.473 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.488+0000 7fa536ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:30.473 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.488+0000 7fa536ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:30.473 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.488+0000 7fa536ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:30.473 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.488+0000 7fa536ffd640 10 monclient(hunting): my global_id is 4457 2024-03-29T11:24:30.473 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.488+0000 7fa536ffd640 1 -- 172.21.15.17:0/3449876402 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa540135210 con 0x7fa54013e1e0 2024-03-29T11:24:30.473 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.488+0000 7fa536ffd640 1 -- 172.21.15.17:0/3449876402 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1728359258 0 0) 0x7fa520004eb0 con 0x7fa5400a9750 2024-03-29T11:24:30.473 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.488+0000 7fa536ffd640 1 -- 172.21.15.17:0/3449876402 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5401354f0 con 0x7fa5400a9750 2024-03-29T11:24:30.473 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.488+0000 7fa536ffd640 1 -- 172.21.15.17:0/3449876402 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fa5280017d0 con 0x7fa5400a9750 2024-03-29T11:24:30.473 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.488+0000 7fa536ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:30.473 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.488+0000 7fa536ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:30.473 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.488+0000 7fa536ffd640 10 monclient(hunting): dump: 2024-03-29T11:24:30.473 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:24:30.474 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:30.474 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:30.474 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:30.474 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:30.474 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:30.474 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:24:30.474 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:24:30.474 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:24:30.474 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:30.474 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.488+0000 7fa536ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:30.474 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.488+0000 7fa536ffd640 1 -- 172.21.15.17:0/3449876402 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1096023150 0 0) 0x7fa540135210 con 0x7fa54013e1e0 2024-03-29T11:24:30.474 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.488+0000 7fa536ffd640 1 -- 172.21.15.17:0/3449876402 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa520004eb0 con 0x7fa54013e1e0 2024-03-29T11:24:30.474 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.488+0000 7fa536ffd640 1 -- 172.21.15.17:0/3449876402 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fa538001cd0 con 0x7fa54013e1e0 2024-03-29T11:24:30.474 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.488+0000 7fa536ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:30.474 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.488+0000 7fa536ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:30.475 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.488+0000 7fa536ffd640 10 monclient(hunting): dump: 2024-03-29T11:24:30.475 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:24:30.475 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:30.475 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:30.475 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:30.475 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:30.475 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:30.475 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:24:30.475 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:24:30.475 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:24:30.475 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:30.475 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.488+0000 7fa536ffd640 1 -- 172.21.15.17:0/3449876402 <== mon.2 v1:172.21.15.17:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 903186670 0 0) 0x7fa520004eb0 con 0x7fa54013e1e0 2024-03-29T11:24:30.475 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.488+0000 7fa536ffd640 1 -- 172.21.15.17:0/3449876402 >> v1:172.21.15.94:6789/0 conn(0x7fa5400a9750 legacy=0x7fa540136e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:30.475 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.488+0000 7fa536ffd640 1 -- 172.21.15.17:0/3449876402 >> v1:172.21.15.17:6789/0 conn(0x7fa540071c30 legacy=0x7fa540133a80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:24:30.475 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.489+0000 7fa536ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:24:30.476 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.489+0000 7fa536ffd640 1 monclient: found mon.c 2024-03-29T11:24:30.476 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.489+0000 7fa536ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:30.476 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.489+0000 7fa536ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:24:30.476 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.489+0000 7fa536ffd640 1 -- 172.21.15.17:0/3449876402 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa540133560 con 0x7fa54013e1e0 2024-03-29T11:24:30.476 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.489+0000 7fa536ffd640 10 monclient: _finish_auth 0 2024-03-29T11:24:30.476 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.489+0000 7fa536ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:24:30.476 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.489+0000 7fa536ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:30.476 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.489+0000 7fa544965640 5 monclient: authenticate success, global_id 4457 2024-03-29T11:24:30.476 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.489+0000 7fa544965640 10 monclient: _renew_subs 2024-03-29T11:24:30.476 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.489+0000 7fa544965640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:24:30.476 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.489+0000 7fa544965640 1 -- 172.21.15.17:0/3449876402 --> v1:172.21.15.17:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5401392f0 con 0x7fa54013e1e0 2024-03-29T11:24:30.476 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.489+0000 7fa544965640 10 client.4457.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:30.476 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.489+0000 7fa544965640 10 monclient: _renew_subs 2024-03-29T11:24:30.476 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.489+0000 7fa544965640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:24:30.476 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.489+0000 7fa544965640 1 -- 172.21.15.17:0/3449876402 --> v1:172.21.15.17:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5401de1a0 con 0x7fa54013e1e0 2024-03-29T11:24:30.477 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.489+0000 7fa544965640 1 librados: init done 2024-03-29T11:24:30.477 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.489+0000 7fa536ffd640 1 -- 172.21.15.17:0/3449876402 <== mon.2 v1:172.21.15.17:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa538003990 con 0x7fa54013e1e0 2024-03-29T11:24:30.477 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.489+0000 7fa536ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:30.477 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.489+0000 7fa536ffd640 1 -- 172.21.15.17:0/3449876402 <== mon.2 v1:172.21.15.17:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fa538002100 con 0x7fa54013e1e0 2024-03-29T11:24:30.477 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.489+0000 7fa536ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:30.477 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.489+0000 7fa536ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:30.477 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.489+0000 7fa536ffd640 10 monclient: dump: 2024-03-29T11:24:30.477 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:24:30.477 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:30.477 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:30.477 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:30.477 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:30.477 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:30.477 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:24:30.478 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:24:30.478 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:24:30.478 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:30.478 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.489+0000 7fa53d59a640 4 set_mon_vals no callback set 2024-03-29T11:24:30.479 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.491+0000 7fa544965640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:30.479 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.491+0000 7fa544965640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:30.479 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.491+0000 7fa544965640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:24:30.479 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.491+0000 7fa544965640 1 -- 172.21.15.17:0/3449876402 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5400abb00 con 0x7fa54013e1e0 2024-03-29T11:24:30.479 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.491+0000 7fa536ffd640 1 -- 172.21.15.17:0/3449876402 <== mon.2 v1:172.21.15.17:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7fa538003d60 con 0x7fa54013e1e0 2024-03-29T11:24:30.480 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.492+0000 7fa536ffd640 1 -- 172.21.15.17:0/3449876402 <== mon.2 v1:172.21.15.17:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7fa538083ba0 con 0x7fa54013e1e0 2024-03-29T11:24:30.480 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.492+0000 7fa536ffd640 10 client.4457.objecter ms_dispatch 0x7fa540000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:30.480 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.492+0000 7fa536ffd640 3 client.4457.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:30.480 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.492+0000 7fa536ffd640 3 client.4457.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:30.480 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.492+0000 7fa536ffd640 20 client.4457.objecter dump_active .. 0 homeless 2024-03-29T11:24:30.480 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.492+0000 7fa53e59c640 10 monclient: get_auth_request con 0x7fa52006df60 auth_method 0 2024-03-29T11:24:30.480 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.493+0000 7fa536ffd640 10 client.4457.objecter ms_handle_connect 0x7fa52006df60 2024-03-29T11:24:30.480 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.498+0000 7fa536ffd640 1 -- 172.21.15.17:0/3449876402 <== mon.2 v1:172.21.15.17:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa53804f480 con 0x7fa54013e1e0 2024-03-29T11:24:30.480 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.498+0000 7fa536ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:30.480 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.498+0000 7fa536ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:30.805 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.823+0000 7fa544965640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:30.805 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.823+0000 7fa544965640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:30.805 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.823+0000 7fa544965640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:24:30.805 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.823+0000 7fa544965640 1 -- 172.21.15.17:0/3449876402 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5401337a0 con 0x7fa54013e1e0 2024-03-29T11:24:30.806 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.824+0000 7fa536ffd640 1 -- 172.21.15.17:0/3449876402 <== mon.2 v1:172.21.15.17:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7fa5401337a0 con 0x7fa54013e1e0 2024-03-29T11:24:30.806 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.824+0000 7fa536ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:30.806 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.824+0000 7fa536ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:30.806 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:24:30.806 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:30.810 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.828+0000 7fa544965640 10 librados: watch_flush enter 2024-03-29T11:24:30.810 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.828+0000 7fa544965640 10 librados: watch_flush exit 2024-03-29T11:24:30.811 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.828+0000 7fa544965640 20 client.4457.objecter shutdown clearing up homeless session... 2024-03-29T11:24:30.811 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.828+0000 7fa544965640 10 client.4457.objecter successfully canceled tick 2024-03-29T11:24:30.811 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.828+0000 7fa544965640 1 -- 172.21.15.17:0/3449876402 >> v1:172.21.15.17:6816/3236185052 conn(0x7fa52006df60 legacy=0x7fa520070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:30.811 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.828+0000 7fa544965640 10 monclient: shutdown 2024-03-29T11:24:30.811 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.828+0000 7fa544965640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:30.811 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.828+0000 7fa544965640 1 -- 172.21.15.17:0/3449876402 >> v1:172.21.15.17:6790/0 conn(0x7fa54013e1e0 legacy=0x7fa540135f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:30.811 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.828+0000 7fa544965640 1 -- 172.21.15.17:0/3449876402 shutdown_connections 2024-03-29T11:24:30.811 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.828+0000 7fa544965640 1 -- 172.21.15.17:0/3449876402 >> 172.21.15.17:0/3449876402 conn(0x7fa54006b660 msgr2=0x7fa54013aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:30.811 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.828+0000 7fa544965640 1 -- 172.21.15.17:0/3449876402 shutdown_connections 2024-03-29T11:24:30.811 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.828+0000 7fa544965640 1 -- 172.21.15.17:0/3449876402 wait complete. 2024-03-29T11:24:30.811 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:30.828+0000 7fa544965640 1 librados: shutdown 2024-03-29T11:24:30.825 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:37.632 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:37.767 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.781+0000 7f4288ef4640 10 monclient: get_monmap_and_config 2024-03-29T11:24:37.767 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.781+0000 7f4288ef4640 10 monclient: build_initial_monmap 2024-03-29T11:24:37.767 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.781+0000 7f4288ef4640 10 monclient: monmap: 2024-03-29T11:24:37.767 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:24:37.767 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:37.767 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:24:37.782678+0000 2024-03-29T11:24:37.767 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:24:37.782678+0000 2024-03-29T11:24:37.767 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:37.767 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:37.767 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:24:37.767 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:24:37.768 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:24:37.768 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:37.768 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.782+0000 7f4288ef4640 1 Processor -- start 2024-03-29T11:24:37.768 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.782+0000 7f4288ef4640 1 -- start start 2024-03-29T11:24:37.768 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.782+0000 7f4288ef4640 10 monclient: init 2024-03-29T11:24:37.768 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.783+0000 7f4288ef4640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:37.768 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.783+0000 7f4288ef4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:24:37.768 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.783+0000 7f4288ef4640 10 monclient(hunting): picked mon.noname-c con 0x7f42841457f0 addr v1:172.21.15.94:6789/0 2024-03-29T11:24:37.768 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.783+0000 7f4288ef4640 10 monclient(hunting): picked mon.noname-b con 0x7f428413f650 addr v1:172.21.15.17:6790/0 2024-03-29T11:24:37.768 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.783+0000 7f4288ef4640 10 monclient(hunting): picked mon.noname-a con 0x7f42840a9d70 addr v1:172.21.15.17:6789/0 2024-03-29T11:24:37.768 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.783+0000 7f4288ef4640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42840aaef0 con 0x7f42840a9d70 2024-03-29T11:24:37.768 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.783+0000 7f4288ef4640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4284145bd0 con 0x7f42841457f0 2024-03-29T11:24:37.768 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.783+0000 7f4288ef4640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4284145eb0 con 0x7f428413f650 2024-03-29T11:24:37.768 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.783+0000 7f4288ef4640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:37.768 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.783+0000 7f4288ef4640 10 monclient(hunting): authenticate will time out at 1183.988390s 2024-03-29T11:24:37.769 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.783+0000 7f428259c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f42841457f0 0x7f428413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:35422/0 (socket says 172.21.15.17:35422) 2024-03-29T11:24:37.769 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.783+0000 7f428259c640 1 -- 172.21.15.17:0/986471901 learned_addr learned my addr 172.21.15.17:0/986471901 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:24:37.769 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.784+0000 7f428159a640 1 -- 172.21.15.17:0/986471901 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1097627301 0 0) 0x7f4284145eb0 con 0x7f428413f650 2024-03-29T11:24:37.769 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.784+0000 7f428159a640 10 monclient(hunting): _negotiate 2024-03-29T11:24:37.769 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.784+0000 7f428159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:37.769 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.784+0000 7f428159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:37.769 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.784+0000 7f428159a640 10 monclient(hunting): my global_id is 4460 2024-03-29T11:24:37.769 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.784+0000 7f428159a640 1 -- 172.21.15.17:0/986471901 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f426c003520 con 0x7f428413f650 2024-03-29T11:24:37.769 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.784+0000 7f428159a640 1 -- 172.21.15.17:0/986471901 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2557732051 0 0) 0x7f426c003520 con 0x7f428413f650 2024-03-29T11:24:37.769 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.784+0000 7f428159a640 1 -- 172.21.15.17:0/986471901 >> v1:172.21.15.94:6789/0 conn(0x7f42841457f0 legacy=0x7f428413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:37.769 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.784+0000 7f428159a640 1 -- 172.21.15.17:0/986471901 >> v1:172.21.15.17:6789/0 conn(0x7f42840a9d70 legacy=0x7f428414d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:24:37.771 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.787+0000 7f428159a640 10 monclient: _finish_hunting 0 2024-03-29T11:24:37.772 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.787+0000 7f428159a640 1 monclient: found mon.noname-b 2024-03-29T11:24:37.772 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.787+0000 7f428159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:37.772 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.787+0000 7f428159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.17:6790/0 2024-03-29T11:24:37.772 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.787+0000 7f428159a640 1 -- 172.21.15.17:0/986471901 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4284146340 con 0x7f428413f650 2024-03-29T11:24:37.772 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.787+0000 7f428159a640 10 monclient: _finish_auth 0 2024-03-29T11:24:37.772 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.787+0000 7f428159a640 10 monclient: _check_auth_tickets 2024-03-29T11:24:37.772 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.787+0000 7f428159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:37.772 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.787+0000 7f428159a640 1 -- 172.21.15.17:0/986471901 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f4278001670 con 0x7f428413f650 2024-03-29T11:24:37.772 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.787+0000 7f428159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:37.772 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.787+0000 7f428159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:37.772 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.787+0000 7f428159a640 10 monclient: dump: 2024-03-29T11:24:37.772 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:24:37.772 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:37.772 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:37.773 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:37.773 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:37.773 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:37.773 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:24:37.773 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:24:37.773 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:24:37.773 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:37.773 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.787+0000 7f4288ef4640 5 monclient: authenticate success, global_id 4460 2024-03-29T11:24:37.773 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.787+0000 7f4288ef4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:37.773 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.787+0000 7f428159a640 1 -- 172.21.15.17:0/986471901 <== mon.2 v1:172.21.15.17:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4278001900 con 0x7f428413f650 2024-03-29T11:24:37.773 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.787+0000 7f428159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:37.773 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.787+0000 7f428159a640 1 -- 172.21.15.17:0/986471901 <== mon.2 v1:172.21.15.17:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f4278001cd0 con 0x7f428413f650 2024-03-29T11:24:37.773 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.787+0000 7f428159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:37.773 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.787+0000 7f428159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:37.774 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.787+0000 7f428159a640 10 monclient: dump: 2024-03-29T11:24:37.774 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:24:37.774 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:37.774 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:37.774 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:37.774 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:37.774 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:37.774 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:24:37.774 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:24:37.774 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:24:37.774 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:37.774 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.788+0000 7f4288ef4640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:37.774 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.788+0000 7f4288ef4640 4 set_mon_vals no callback set 2024-03-29T11:24:37.774 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.789+0000 7f4288ef4640 10 monclient: shutdown 2024-03-29T11:24:37.775 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.789+0000 7f4288ef4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:37.775 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.789+0000 7f4288ef4640 1 -- 172.21.15.17:0/986471901 >> v1:172.21.15.17:6790/0 conn(0x7f428413f650 legacy=0x7f42840b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:37.775 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.789+0000 7f4288ef4640 1 -- 172.21.15.17:0/986471901 shutdown_connections 2024-03-29T11:24:37.775 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.789+0000 7f4288ef4640 1 -- 172.21.15.17:0/986471901 >> 172.21.15.17:0/986471901 conn(0x7f4284066a00 msgr2=0x7f4284066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:37.775 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.789+0000 7f4288ef4640 1 -- 172.21.15.17:0/986471901 shutdown_connections 2024-03-29T11:24:37.775 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.789+0000 7f4288ef4640 1 -- 172.21.15.17:0/986471901 wait complete. 2024-03-29T11:24:37.779 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.791+0000 7f4288ef4640 10 monclient: build_initial_monmap 2024-03-29T11:24:37.779 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.791+0000 7f4288ef4640 10 monclient: monmap: 2024-03-29T11:24:37.779 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:24:37.779 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:37.779 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:24:37.779 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:24:37.779 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:37.780 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:37.780 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:24:37.780 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:24:37.780 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:24:37.780 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:37.780 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.791+0000 7f4288ef4640 1 librados: starting msgr at 2024-03-29T11:24:37.780 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.791+0000 7f4288ef4640 1 librados: starting objecter 2024-03-29T11:24:37.780 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.791+0000 7f4288ef4640 1 Processor -- start 2024-03-29T11:24:37.780 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.791+0000 7f4288ef4640 1 -- start start 2024-03-29T11:24:37.780 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.791+0000 7f4288ef4640 1 librados: setting wanted keys 2024-03-29T11:24:37.780 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.791+0000 7f4288ef4640 1 librados: calling monclient init 2024-03-29T11:24:37.780 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.791+0000 7f4288ef4640 10 monclient: init 2024-03-29T11:24:37.780 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.792+0000 7f4288ef4640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:37.780 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.792+0000 7f4288ef4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:24:37.781 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.792+0000 7f4288ef4640 10 monclient(hunting): picked mon.noname-b con 0x7f42840a9d70 addr v1:172.21.15.94:6789/0 2024-03-29T11:24:37.781 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.792+0000 7f4288ef4640 10 monclient(hunting): picked mon.noname-c con 0x7f428413f650 addr v1:172.21.15.17:6790/0 2024-03-29T11:24:37.781 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.792+0000 7f4288ef4640 10 monclient(hunting): picked mon.noname-a con 0x7f42841457f0 addr v1:172.21.15.17:6789/0 2024-03-29T11:24:37.781 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.792+0000 7f4288ef4640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42841dfdd0 con 0x7f42841457f0 2024-03-29T11:24:37.781 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.792+0000 7f4288ef4640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4284142090 con 0x7f42840a9d70 2024-03-29T11:24:37.781 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.792+0000 7f4288ef4640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4284142370 con 0x7f428413f650 2024-03-29T11:24:37.781 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.792+0000 7f4288ef4640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:37.781 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.792+0000 7f4288ef4640 10 monclient(hunting): authenticate will time out at 1183.997829s 2024-03-29T11:24:37.781 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.793+0000 7f4281d9b640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f428413f650 0x7f42841defb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:51780/0 (socket says 172.21.15.17:51780) 2024-03-29T11:24:37.781 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.793+0000 7f4281d9b640 1 -- 172.21.15.17:0/2223501776 learned_addr learned my addr 172.21.15.17:0/2223501776 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:24:37.781 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.793+0000 7f4272ffd640 10 client.?.objecter ms_handle_connect 0x7f428413f650 2024-03-29T11:24:37.781 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.793+0000 7f4272ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:37.781 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.794+0000 7f4272ffd640 1 -- 172.21.15.17:0/2223501776 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2981280767 0 0) 0x7f4284142370 con 0x7f428413f650 2024-03-29T11:24:37.781 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.794+0000 7f4272ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:37.782 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.794+0000 7f4272ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:37.782 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.794+0000 7f4272ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:37.782 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.794+0000 7f4272ffd640 10 monclient(hunting): my global_id is 4463 2024-03-29T11:24:37.782 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.794+0000 7f4272ffd640 1 -- 172.21.15.17:0/2223501776 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4264004eb0 con 0x7f428413f650 2024-03-29T11:24:37.782 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.794+0000 7f4272ffd640 10 client.?.objecter ms_handle_connect 0x7f42840a9d70 2024-03-29T11:24:37.782 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.794+0000 7f4272ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:37.782 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.794+0000 7f4272ffd640 1 -- 172.21.15.17:0/2223501776 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2019201502 0 0) 0x7f4264004eb0 con 0x7f428413f650 2024-03-29T11:24:37.782 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.794+0000 7f4272ffd640 1 -- 172.21.15.17:0/2223501776 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4284142370 con 0x7f428413f650 2024-03-29T11:24:37.782 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.794+0000 7f4272ffd640 1 -- 172.21.15.17:0/2223501776 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f4278001630 con 0x7f428413f650 2024-03-29T11:24:37.782 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.794+0000 7f4272ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:37.782 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.794+0000 7f4272ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:37.782 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.794+0000 7f4272ffd640 10 monclient(hunting): dump: 2024-03-29T11:24:37.782 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:24:37.782 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:37.783 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:37.783 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:37.783 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:37.783 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:37.783 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:24:37.783 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:24:37.783 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:24:37.783 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:37.783 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.794+0000 7f4272ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:37.783 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.794+0000 7f4272ffd640 1 -- 172.21.15.17:0/2223501776 <== mon.2 v1:172.21.15.17:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2612153420 0 0) 0x7f4284142370 con 0x7f428413f650 2024-03-29T11:24:37.783 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.794+0000 7f4272ffd640 1 -- 172.21.15.17:0/2223501776 >> v1:172.21.15.94:6789/0 conn(0x7f42840a9d70 legacy=0x7f4284142790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:37.783 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.794+0000 7f4272ffd640 1 -- 172.21.15.17:0/2223501776 >> v1:172.21.15.17:6789/0 conn(0x7f42841457f0 legacy=0x7f42841df6c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:24:37.783 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.795+0000 7f4272ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:24:37.783 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.795+0000 7f4272ffd640 1 monclient: found mon.c 2024-03-29T11:24:37.783 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.795+0000 7f4272ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:37.784 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.795+0000 7f4272ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:24:37.784 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.795+0000 7f4272ffd640 1 -- 172.21.15.17:0/2223501776 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4284135e00 con 0x7f428413f650 2024-03-29T11:24:37.784 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.795+0000 7f4272ffd640 10 monclient: _finish_auth 0 2024-03-29T11:24:37.784 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.795+0000 7f4272ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:24:37.784 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.795+0000 7f4272ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:37.784 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.795+0000 7f4288ef4640 5 monclient: authenticate success, global_id 4463 2024-03-29T11:24:37.784 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.795+0000 7f4288ef4640 10 monclient: _renew_subs 2024-03-29T11:24:37.784 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.795+0000 7f4288ef4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:24:37.784 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.795+0000 7f4288ef4640 1 -- 172.21.15.17:0/2223501776 --> v1:172.21.15.17:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42841405a0 con 0x7f428413f650 2024-03-29T11:24:37.784 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.795+0000 7f4288ef4640 10 client.4463.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:37.784 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.795+0000 7f4288ef4640 10 monclient: _renew_subs 2024-03-29T11:24:37.784 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.795+0000 7f4288ef4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:24:37.784 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.795+0000 7f4288ef4640 1 -- 172.21.15.17:0/2223501776 --> v1:172.21.15.17:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f42841412c0 con 0x7f428413f650 2024-03-29T11:24:37.784 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.795+0000 7f4288ef4640 1 librados: init done 2024-03-29T11:24:37.784 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.796+0000 7f4272ffd640 1 -- 172.21.15.17:0/2223501776 <== mon.2 v1:172.21.15.17:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42780039a0 con 0x7f428413f650 2024-03-29T11:24:37.785 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.796+0000 7f4272ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:37.785 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.796+0000 7f4272ffd640 1 -- 172.21.15.17:0/2223501776 <== mon.2 v1:172.21.15.17:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f4278003d70 con 0x7f428413f650 2024-03-29T11:24:37.785 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.796+0000 7f4272ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:37.785 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.796+0000 7f4272ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:37.785 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.796+0000 7f4280d99640 4 set_mon_vals no callback set 2024-03-29T11:24:37.785 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.796+0000 7f4272ffd640 10 monclient: dump: 2024-03-29T11:24:37.785 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:24:37.785 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:37.785 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:37.785 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:37.785 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:37.785 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:37.785 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:24:37.785 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:24:37.786 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:24:37.786 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:37.787 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.798+0000 7f4288ef4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:37.787 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.798+0000 7f4288ef4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:37.787 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.798+0000 7f4288ef4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:24:37.787 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.798+0000 7f4288ef4640 1 -- 172.21.15.17:0/2223501776 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4268005080 con 0x7f428413f650 2024-03-29T11:24:37.787 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.798+0000 7f4272ffd640 1 -- 172.21.15.17:0/2223501776 <== mon.2 v1:172.21.15.17:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f4278003ed0 con 0x7f428413f650 2024-03-29T11:24:37.787 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.799+0000 7f4272ffd640 1 -- 172.21.15.17:0/2223501776 <== mon.2 v1:172.21.15.17:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f4278084d20 con 0x7f428413f650 2024-03-29T11:24:37.787 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.799+0000 7f4272ffd640 10 client.4463.objecter ms_dispatch 0x7f4284000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:37.788 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.799+0000 7f4272ffd640 3 client.4463.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:37.788 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.799+0000 7f4272ffd640 3 client.4463.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:37.788 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.799+0000 7f4272ffd640 20 client.4463.objecter dump_active .. 0 homeless 2024-03-29T11:24:37.788 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.799+0000 7f428259c640 10 monclient: get_auth_request con 0x7f426406dc20 auth_method 0 2024-03-29T11:24:37.788 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.800+0000 7f4272ffd640 10 client.4463.objecter ms_handle_connect 0x7f426406dc20 2024-03-29T11:24:37.788 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.806+0000 7f4272ffd640 1 -- 172.21.15.17:0/2223501776 <== mon.2 v1:172.21.15.17:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4278050600 con 0x7f428413f650 2024-03-29T11:24:37.788 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.806+0000 7f4272ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:37.789 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:37.806+0000 7f4272ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:38.104 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:38.121+0000 7f4288ef4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:38.104 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:38.121+0000 7f4288ef4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:38.104 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:38.121+0000 7f4288ef4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:24:38.104 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:38.121+0000 7f4288ef4640 1 -- 172.21.15.17:0/2223501776 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4268005330 con 0x7f428413f650 2024-03-29T11:24:38.104 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:38.122+0000 7f4272ffd640 1 -- 172.21.15.17:0/2223501776 <== mon.2 v1:172.21.15.17:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f4278052530 con 0x7f428413f650 2024-03-29T11:24:38.105 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:38.122+0000 7f4272ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:38.105 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:38.122+0000 7f4272ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:38.105 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:24:38.105 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:38.108 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:38.125+0000 7f4288ef4640 10 librados: watch_flush enter 2024-03-29T11:24:38.108 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:38.125+0000 7f4288ef4640 10 librados: watch_flush exit 2024-03-29T11:24:38.108 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:38.125+0000 7f4288ef4640 20 client.4463.objecter shutdown clearing up homeless session... 2024-03-29T11:24:38.108 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:38.125+0000 7f4288ef4640 10 client.4463.objecter successfully canceled tick 2024-03-29T11:24:38.108 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:38.126+0000 7f4288ef4640 1 -- 172.21.15.17:0/2223501776 >> v1:172.21.15.17:6816/3236185052 conn(0x7f426406dc20 legacy=0x7f42640700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:38.108 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:38.126+0000 7f4288ef4640 10 monclient: shutdown 2024-03-29T11:24:38.108 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:38.126+0000 7f4288ef4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:38.108 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:38.126+0000 7f4288ef4640 1 -- 172.21.15.17:0/2223501776 >> v1:172.21.15.17:6790/0 conn(0x7f428413f650 legacy=0x7f42841defb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:38.109 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:38.126+0000 7f4288ef4640 1 -- 172.21.15.17:0/2223501776 shutdown_connections 2024-03-29T11:24:38.109 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:38.126+0000 7f4288ef4640 1 -- 172.21.15.17:0/2223501776 >> 172.21.15.17:0/2223501776 conn(0x7f4284066a00 msgr2=0x7f42840b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:38.109 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:38.126+0000 7f4288ef4640 1 -- 172.21.15.17:0/2223501776 shutdown_connections 2024-03-29T11:24:38.109 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:38.126+0000 7f4288ef4640 1 -- 172.21.15.17:0/2223501776 wait complete. 2024-03-29T11:24:38.109 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:38.126+0000 7f4288ef4640 1 librados: shutdown 2024-03-29T11:24:38.122 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:44.929 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:45.067 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.084+0000 7f64b48ae640 10 monclient: get_monmap_and_config 2024-03-29T11:24:45.067 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.084+0000 7f64b48ae640 10 monclient: build_initial_monmap 2024-03-29T11:24:45.067 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.084+0000 7f64b48ae640 10 monclient: monmap: 2024-03-29T11:24:45.067 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:24:45.067 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:45.067 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:24:45.085394+0000 2024-03-29T11:24:45.067 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:24:45.085394+0000 2024-03-29T11:24:45.067 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:45.067 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:45.067 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:24:45.067 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:24:45.067 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:24:45.068 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:45.069 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.085+0000 7f64b48ae640 1 Processor -- start 2024-03-29T11:24:45.069 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.085+0000 7f64b48ae640 1 -- start start 2024-03-29T11:24:45.069 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.085+0000 7f64b48ae640 10 monclient: init 2024-03-29T11:24:45.069 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.086+0000 7f64b48ae640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:45.069 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.086+0000 7f64b48ae640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:24:45.069 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.086+0000 7f64b48ae640 10 monclient(hunting): picked mon.noname-b con 0x7f64b00af290 addr v1:172.21.15.17:6790/0 2024-03-29T11:24:45.069 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.086+0000 7f64b48ae640 10 monclient(hunting): picked mon.noname-c con 0x7f64b00ae740 addr v1:172.21.15.94:6789/0 2024-03-29T11:24:45.069 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.086+0000 7f64b48ae640 10 monclient(hunting): picked mon.noname-a con 0x7f64b00af670 addr v1:172.21.15.17:6789/0 2024-03-29T11:24:45.069 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.086+0000 7f64b48ae640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b00b0980 con 0x7f64b00af670 2024-03-29T11:24:45.070 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.086+0000 7f64b48ae640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b00b39f0 con 0x7f64b00ae740 2024-03-29T11:24:45.070 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.086+0000 7f64b48ae640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b00aebd0 con 0x7f64b00af290 2024-03-29T11:24:45.070 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.086+0000 7f64b48ae640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:45.070 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.086+0000 7f64b48ae640 10 monclient(hunting): authenticate will time out at 1191.291357s 2024-03-29T11:24:45.070 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.086+0000 7f64ae59c640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f64b00af290 0x7f64b00b4430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:56874/0 (socket says 172.21.15.17:56874) 2024-03-29T11:24:45.070 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.086+0000 7f64ae59c640 1 -- 172.21.15.17:0/434782133 learned_addr learned my addr 172.21.15.17:0/434782133 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:24:45.070 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.086+0000 7f64ad59a640 1 -- 172.21.15.17:0/434782133 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1756705100 0 0) 0x7f64b00aebd0 con 0x7f64b00af290 2024-03-29T11:24:45.070 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.086+0000 7f64ad59a640 10 monclient(hunting): _negotiate 2024-03-29T11:24:45.070 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.086+0000 7f64ad59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:45.070 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.086+0000 7f64ad59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:45.070 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.086+0000 7f64ad59a640 10 monclient(hunting): my global_id is 4466 2024-03-29T11:24:45.070 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.086+0000 7f64ad59a640 1 -- 172.21.15.17:0/434782133 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6498003520 con 0x7f64b00af290 2024-03-29T11:24:45.071 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.087+0000 7f64ad59a640 1 -- 172.21.15.17:0/434782133 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2516048663 0 0) 0x7f6498003520 con 0x7f64b00af290 2024-03-29T11:24:45.071 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.087+0000 7f64ad59a640 1 -- 172.21.15.17:0/434782133 >> v1:172.21.15.94:6789/0 conn(0x7f64b00ae740 legacy=0x7f64b01439f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:45.071 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.087+0000 7f64ad59a640 1 -- 172.21.15.17:0/434782133 >> v1:172.21.15.17:6789/0 conn(0x7f64b00af670 legacy=0x7f64b0151e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:24:45.073 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.090+0000 7f64ad59a640 10 monclient: _finish_hunting 0 2024-03-29T11:24:45.073 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.090+0000 7f64ad59a640 1 monclient: found mon.noname-b 2024-03-29T11:24:45.073 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.090+0000 7f64ad59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:45.073 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.090+0000 7f64ad59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.17:6790/0 2024-03-29T11:24:45.074 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.090+0000 7f64ad59a640 1 -- 172.21.15.17:0/434782133 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b00b4b40 con 0x7f64b00af290 2024-03-29T11:24:45.074 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.090+0000 7f64ad59a640 10 monclient: _finish_auth 0 2024-03-29T11:24:45.074 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.090+0000 7f64ad59a640 10 monclient: _check_auth_tickets 2024-03-29T11:24:45.074 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.090+0000 7f64ad59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:45.074 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.090+0000 7f64ad59a640 1 -- 172.21.15.17:0/434782133 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f64a0001a10 con 0x7f64b00af290 2024-03-29T11:24:45.074 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.090+0000 7f64ad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:45.074 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.090+0000 7f64ad59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:45.074 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.090+0000 7f64ad59a640 10 monclient: dump: 2024-03-29T11:24:45.074 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:24:45.074 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:45.074 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:45.074 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:45.074 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:45.074 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:45.075 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:24:45.075 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:24:45.075 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:24:45.075 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:45.075 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.090+0000 7f64b48ae640 5 monclient: authenticate success, global_id 4466 2024-03-29T11:24:45.075 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.090+0000 7f64b48ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:45.075 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.090+0000 7f64ad59a640 1 -- 172.21.15.17:0/434782133 <== mon.2 v1:172.21.15.17:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64a0001be0 con 0x7f64b00af290 2024-03-29T11:24:45.075 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.090+0000 7f64ad59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:45.075 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.090+0000 7f64ad59a640 1 -- 172.21.15.17:0/434782133 <== mon.2 v1:172.21.15.17:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f64a0002030 con 0x7f64b00af290 2024-03-29T11:24:45.075 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.090+0000 7f64ad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:45.075 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.090+0000 7f64ad59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:45.075 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.090+0000 7f64ad59a640 10 monclient: dump: 2024-03-29T11:24:45.075 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:24:45.075 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:45.075 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:45.076 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:45.076 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:45.076 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:45.076 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:24:45.076 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:24:45.076 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:24:45.076 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:45.076 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.090+0000 7f64b48ae640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:45.076 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.091+0000 7f64b48ae640 4 set_mon_vals no callback set 2024-03-29T11:24:45.078 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.091+0000 7f64b48ae640 10 monclient: shutdown 2024-03-29T11:24:45.078 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.092+0000 7f64b48ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:45.078 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.092+0000 7f64b48ae640 1 -- 172.21.15.17:0/434782133 >> v1:172.21.15.17:6790/0 conn(0x7f64b00af290 legacy=0x7f64b00b4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:45.078 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.092+0000 7f64b48ae640 1 -- 172.21.15.17:0/434782133 shutdown_connections 2024-03-29T11:24:45.078 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.092+0000 7f64b48ae640 1 -- 172.21.15.17:0/434782133 >> 172.21.15.17:0/434782133 conn(0x7f64b0066a00 msgr2=0x7f64b0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:45.078 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.092+0000 7f64b48ae640 1 -- 172.21.15.17:0/434782133 shutdown_connections 2024-03-29T11:24:45.079 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.092+0000 7f64b48ae640 1 -- 172.21.15.17:0/434782133 wait complete. 2024-03-29T11:24:45.079 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.093+0000 7f64b48ae640 10 monclient: build_initial_monmap 2024-03-29T11:24:45.079 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.093+0000 7f64b48ae640 10 monclient: monmap: 2024-03-29T11:24:45.079 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:24:45.079 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:45.079 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:24:45.079 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:24:45.079 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:45.079 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:45.079 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:24:45.079 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:24:45.079 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:24:45.079 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:45.079 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.094+0000 7f64b48ae640 1 librados: starting msgr at 2024-03-29T11:24:45.080 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.094+0000 7f64b48ae640 1 librados: starting objecter 2024-03-29T11:24:45.080 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.094+0000 7f64b48ae640 1 Processor -- start 2024-03-29T11:24:45.080 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.094+0000 7f64b48ae640 1 -- start start 2024-03-29T11:24:45.080 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.094+0000 7f64b48ae640 1 librados: setting wanted keys 2024-03-29T11:24:45.080 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.094+0000 7f64b48ae640 1 librados: calling monclient init 2024-03-29T11:24:45.080 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.094+0000 7f64b48ae640 10 monclient: init 2024-03-29T11:24:45.080 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.095+0000 7f64b48ae640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:45.080 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.095+0000 7f64b48ae640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:24:45.080 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.096+0000 7f64b48ae640 10 monclient(hunting): picked mon.noname-b con 0x7f64b00af290 addr v1:172.21.15.94:6789/0 2024-03-29T11:24:45.080 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.096+0000 7f64b48ae640 10 monclient(hunting): picked mon.noname-c con 0x7f64b00af670 addr v1:172.21.15.17:6790/0 2024-03-29T11:24:45.082 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.096+0000 7f64add9b640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f64b00af670 0x7f64b0146c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:58162/0 (socket says 172.21.15.17:58162) 2024-03-29T11:24:45.082 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.097+0000 7f64ae59c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f64b00af290 0x7f64b0071180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:40966/0 (socket says 172.21.15.17:40966) 2024-03-29T11:24:45.082 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.098+0000 7f64b48ae640 10 monclient(hunting): picked mon.noname-a con 0x7f64b0143260 addr v1:172.21.15.17:6789/0 2024-03-29T11:24:45.083 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.098+0000 7f64b48ae640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b01451e0 con 0x7f64b0143260 2024-03-29T11:24:45.083 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.098+0000 7f64b48ae640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b0144910 con 0x7f64b00af290 2024-03-29T11:24:45.083 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.098+0000 7f64b48ae640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b01db440 con 0x7f64b00af670 2024-03-29T11:24:45.083 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.098+0000 7f64b48ae640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:45.083 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.098+0000 7f64b48ae640 10 monclient(hunting): authenticate will time out at 1191.303161s 2024-03-29T11:24:45.083 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.098+0000 7f64add9b640 1 -- 172.21.15.17:0/4106165224 learned_addr learned my addr 172.21.15.17:0/4106165224 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:24:45.083 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.098+0000 7f649effd640 10 client.?.objecter ms_handle_connect 0x7f64b00af670 2024-03-29T11:24:45.083 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.098+0000 7f649effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:45.083 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.098+0000 7f649effd640 1 -- 172.21.15.17:0/4106165224 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 566171987 0 0) 0x7f64b01db440 con 0x7f64b00af670 2024-03-29T11:24:45.083 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.098+0000 7f649effd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:45.083 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.098+0000 7f649effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:45.083 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.098+0000 7f649effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:45.083 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.098+0000 7f649effd640 10 monclient(hunting): my global_id is 4469 2024-03-29T11:24:45.083 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.098+0000 7f649effd640 1 -- 172.21.15.17:0/4106165224 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6490004eb0 con 0x7f64b00af670 2024-03-29T11:24:45.084 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.098+0000 7f649effd640 1 -- 172.21.15.17:0/4106165224 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3569234989 0 0) 0x7f6490004eb0 con 0x7f64b00af670 2024-03-29T11:24:45.084 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.098+0000 7f649effd640 1 -- 172.21.15.17:0/4106165224 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64b01db440 con 0x7f64b00af670 2024-03-29T11:24:45.084 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.098+0000 7f649effd640 1 -- 172.21.15.17:0/4106165224 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f64a4001a30 con 0x7f64b00af670 2024-03-29T11:24:45.084 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.098+0000 7f649effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:45.084 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.098+0000 7f649effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:45.084 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.098+0000 7f649effd640 10 monclient(hunting): dump: 2024-03-29T11:24:45.084 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:24:45.084 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:45.084 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:45.084 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:45.084 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:45.084 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:45.084 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:24:45.084 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:24:45.084 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:24:45.085 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:45.085 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.098+0000 7f649effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:45.085 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.099+0000 7f649effd640 1 -- 172.21.15.17:0/4106165224 <== mon.2 v1:172.21.15.17:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1995434025 0 0) 0x7f64b01db440 con 0x7f64b00af670 2024-03-29T11:24:45.085 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.099+0000 7f649effd640 1 -- 172.21.15.17:0/4106165224 >> v1:172.21.15.94:6789/0 conn(0x7f64b00af290 legacy=0x7f64b0071180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:45.085 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.099+0000 7f649effd640 1 -- 172.21.15.17:0/4106165224 >> v1:172.21.15.17:6789/0 conn(0x7f64b0143260 legacy=0x7f64b01e6ca0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:24:45.085 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.099+0000 7f649effd640 10 monclient: _finish_hunting 0 2024-03-29T11:24:45.085 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.099+0000 7f649effd640 1 monclient: found mon.c 2024-03-29T11:24:45.085 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.099+0000 7f649effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:45.085 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.099+0000 7f649effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:24:45.085 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.099+0000 7f649effd640 1 -- 172.21.15.17:0/4106165224 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b0071890 con 0x7f64b00af670 2024-03-29T11:24:45.085 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.099+0000 7f649effd640 10 monclient: _finish_auth 0 2024-03-29T11:24:45.085 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.099+0000 7f649effd640 10 monclient: _check_auth_tickets 2024-03-29T11:24:45.085 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.099+0000 7f649effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:45.085 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.099+0000 7f64b48ae640 5 monclient: authenticate success, global_id 4469 2024-03-29T11:24:45.085 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.099+0000 7f64b48ae640 10 monclient: _renew_subs 2024-03-29T11:24:45.086 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.099+0000 7f64b48ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:24:45.086 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.099+0000 7f64b48ae640 1 -- 172.21.15.17:0/4106165224 --> v1:172.21.15.17:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64b0071b60 con 0x7f64b00af670 2024-03-29T11:24:45.086 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.099+0000 7f64b48ae640 10 client.4469.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:45.086 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.099+0000 7f64b48ae640 10 monclient: _renew_subs 2024-03-29T11:24:45.086 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.099+0000 7f64b48ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:24:45.086 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.099+0000 7f64b48ae640 1 -- 172.21.15.17:0/4106165224 --> v1:172.21.15.17:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f64b0147580 con 0x7f64b00af670 2024-03-29T11:24:45.086 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.099+0000 7f64b48ae640 1 librados: init done 2024-03-29T11:24:45.086 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.099+0000 7f649effd640 1 -- 172.21.15.17:0/4106165224 <== mon.2 v1:172.21.15.17:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64a4001d30 con 0x7f64b00af670 2024-03-29T11:24:45.086 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.099+0000 7f649effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:45.087 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.099+0000 7f64acd99640 4 set_mon_vals no callback set 2024-03-29T11:24:45.087 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.099+0000 7f649effd640 1 -- 172.21.15.17:0/4106165224 <== mon.2 v1:172.21.15.17:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f64a4003a60 con 0x7f64b00af670 2024-03-29T11:24:45.087 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.099+0000 7f649effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:45.087 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.099+0000 7f649effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:45.087 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.099+0000 7f649effd640 10 monclient: dump: 2024-03-29T11:24:45.087 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:24:45.087 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:45.087 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:45.088 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:45.088 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:45.088 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:45.088 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:24:45.088 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:24:45.088 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:24:45.088 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:45.088 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.100+0000 7f64b48ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:45.088 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.100+0000 7f64b48ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:45.088 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.101+0000 7f64b48ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:24:45.088 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.101+0000 7f649effd640 1 -- 172.21.15.17:0/4106165224 <== mon.2 v1:172.21.15.17:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f64a4003c40 con 0x7f64b00af670 2024-03-29T11:24:45.088 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.101+0000 7f64b48ae640 1 -- 172.21.15.17:0/4106165224 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6494005080 con 0x7f64b00af670 2024-03-29T11:24:45.088 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.102+0000 7f649effd640 1 -- 172.21.15.17:0/4106165224 <== mon.2 v1:172.21.15.17:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f64a4084bd0 con 0x7f64b00af670 2024-03-29T11:24:45.088 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.102+0000 7f649effd640 10 client.4469.objecter ms_dispatch 0x7f64b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:45.088 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.102+0000 7f649effd640 3 client.4469.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:45.089 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.102+0000 7f649effd640 3 client.4469.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:45.089 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.102+0000 7f649effd640 20 client.4469.objecter dump_active .. 0 homeless 2024-03-29T11:24:45.089 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.102+0000 7f64ae59c640 10 monclient: get_auth_request con 0x7f649006dba0 auth_method 0 2024-03-29T11:24:45.089 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.102+0000 7f649effd640 10 client.4469.objecter ms_handle_connect 0x7f649006dba0 2024-03-29T11:24:45.092 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.110+0000 7f649effd640 1 -- 172.21.15.17:0/4106165224 <== mon.2 v1:172.21.15.17:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f64a4050510 con 0x7f64b00af670 2024-03-29T11:24:45.092 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.110+0000 7f649effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:45.092 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.110+0000 7f649effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:45.412 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.430+0000 7f64b48ae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:45.412 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.430+0000 7f64b48ae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:45.412 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.430+0000 7f64b48ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:24:45.412 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.430+0000 7f64b48ae640 1 -- 172.21.15.17:0/4106165224 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6494005330 con 0x7f64b00af670 2024-03-29T11:24:45.413 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.430+0000 7f649effd640 1 -- 172.21.15.17:0/4106165224 <== mon.2 v1:172.21.15.17:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f64a4052440 con 0x7f64b00af670 2024-03-29T11:24:45.413 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.430+0000 7f649effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:45.413 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.430+0000 7f649effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:45.413 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:24:45.413 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:45.417 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.434+0000 7f64b48ae640 10 librados: watch_flush enter 2024-03-29T11:24:45.417 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.434+0000 7f64b48ae640 10 librados: watch_flush exit 2024-03-29T11:24:45.417 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.434+0000 7f64b48ae640 20 client.4469.objecter shutdown clearing up homeless session... 2024-03-29T11:24:45.417 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.434+0000 7f64b48ae640 10 client.4469.objecter successfully canceled tick 2024-03-29T11:24:45.417 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.434+0000 7f64b48ae640 1 -- 172.21.15.17:0/4106165224 >> v1:172.21.15.17:6816/3236185052 conn(0x7f649006dba0 legacy=0x7f6490070060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:45.417 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.434+0000 7f64b48ae640 10 monclient: shutdown 2024-03-29T11:24:45.417 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.434+0000 7f64b48ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:45.417 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.434+0000 7f64b48ae640 1 -- 172.21.15.17:0/4106165224 >> v1:172.21.15.17:6790/0 conn(0x7f64b00af670 legacy=0x7f64b0146c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:45.417 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.434+0000 7f64b48ae640 1 -- 172.21.15.17:0/4106165224 shutdown_connections 2024-03-29T11:24:45.417 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.434+0000 7f64b48ae640 1 -- 172.21.15.17:0/4106165224 >> 172.21.15.17:0/4106165224 conn(0x7f64b0066a00 msgr2=0x7f64b013fa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:45.417 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.434+0000 7f64b48ae640 1 -- 172.21.15.17:0/4106165224 shutdown_connections 2024-03-29T11:24:45.418 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.434+0000 7f64b48ae640 1 -- 172.21.15.17:0/4106165224 wait complete. 2024-03-29T11:24:45.418 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:45.434+0000 7f64b48ae640 1 librados: shutdown 2024-03-29T11:24:45.430 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:52.237 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:52.385 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.402+0000 7f884d152640 10 monclient: get_monmap_and_config 2024-03-29T11:24:52.386 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.402+0000 7f884d152640 10 monclient: build_initial_monmap 2024-03-29T11:24:52.386 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.402+0000 7f884d152640 10 monclient: monmap: 2024-03-29T11:24:52.386 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:24:52.386 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:52.386 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:24:52.403817+0000 2024-03-29T11:24:52.386 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:24:52.403817+0000 2024-03-29T11:24:52.386 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:52.386 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:52.386 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:24:52.386 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:24:52.386 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:24:52.386 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:52.388 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.403+0000 7f884d152640 1 Processor -- start 2024-03-29T11:24:52.388 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.404+0000 7f884d152640 1 -- start start 2024-03-29T11:24:52.388 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.404+0000 7f884d152640 10 monclient: init 2024-03-29T11:24:52.388 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.405+0000 7f884d152640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:52.388 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.405+0000 7f884d152640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:24:52.388 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.405+0000 7f884d152640 10 monclient(hunting): picked mon.noname-c con 0x7f88480b0500 addr v1:172.21.15.94:6789/0 2024-03-29T11:24:52.388 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.405+0000 7f884d152640 10 monclient(hunting): picked mon.noname-a con 0x7f88480b08e0 addr v1:172.21.15.17:6789/0 2024-03-29T11:24:52.388 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.405+0000 7f884d152640 10 monclient(hunting): picked mon.noname-b con 0x7f88480aeca0 addr v1:172.21.15.17:6790/0 2024-03-29T11:24:52.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.405+0000 7f884d152640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8848140620 con 0x7f88480b08e0 2024-03-29T11:24:52.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.405+0000 7f884d152640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f884813daf0 con 0x7f88480b0500 2024-03-29T11:24:52.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.405+0000 7f884d152640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88480ae180 con 0x7f88480aeca0 2024-03-29T11:24:52.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.405+0000 7f884d152640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:52.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.405+0000 7f884d152640 10 monclient(hunting): authenticate will time out at 1198.610058s 2024-03-29T11:24:52.390 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.406+0000 7f884759e640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f88480aeca0 0x7f884814b7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:33392/0 (socket says 172.21.15.17:33392) 2024-03-29T11:24:52.390 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.406+0000 7f884759e640 1 -- 172.21.15.17:0/3533401351 learned_addr learned my addr 172.21.15.17:0/3533401351 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:24:52.390 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.407+0000 7f8845d9b640 1 -- 172.21.15.17:0/3533401351 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3610960492 0 0) 0x7f88480ae180 con 0x7f88480aeca0 2024-03-29T11:24:52.390 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.407+0000 7f8845d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:24:52.390 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.407+0000 7f8845d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:52.394 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.407+0000 7f8845d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:52.394 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.407+0000 7f8845d9b640 10 monclient(hunting): my global_id is 4472 2024-03-29T11:24:52.394 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.407+0000 7f8845d9b640 1 -- 172.21.15.17:0/3533401351 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8834003520 con 0x7f88480aeca0 2024-03-29T11:24:52.394 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.409+0000 7f8845d9b640 1 -- 172.21.15.17:0/3533401351 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1580289832 0 0) 0x7f884813daf0 con 0x7f88480b0500 2024-03-29T11:24:52.395 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.409+0000 7f8845d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:24:52.395 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.409+0000 7f8845d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:52.395 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.409+0000 7f8845d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:52.395 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.409+0000 7f8845d9b640 10 monclient(hunting): my global_id is 4459 2024-03-29T11:24:52.395 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.409+0000 7f8845d9b640 1 -- 172.21.15.17:0/3533401351 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88480ae180 con 0x7f88480b0500 2024-03-29T11:24:52.395 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.409+0000 7f8845d9b640 1 -- 172.21.15.17:0/3533401351 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1390749528 0 0) 0x7f8848140620 con 0x7f88480b08e0 2024-03-29T11:24:52.395 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.409+0000 7f8845d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:24:52.395 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.409+0000 7f8845d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:52.395 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.409+0000 7f8845d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:52.395 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.409+0000 7f8845d9b640 10 monclient(hunting): my global_id is 4425 2024-03-29T11:24:52.395 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.409+0000 7f8845d9b640 1 -- 172.21.15.17:0/3533401351 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f884813daf0 con 0x7f88480b08e0 2024-03-29T11:24:52.395 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.409+0000 7f8845d9b640 1 -- 172.21.15.17:0/3533401351 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1415661632 0 0) 0x7f8834003520 con 0x7f88480aeca0 2024-03-29T11:24:52.395 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.409+0000 7f8845d9b640 1 -- 172.21.15.17:0/3533401351 >> v1:172.21.15.94:6789/0 conn(0x7f88480b0500 legacy=0x7f884813ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:52.395 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.409+0000 7f8845d9b640 1 -- 172.21.15.17:0/3533401351 >> v1:172.21.15.17:6789/0 conn(0x7f88480b08e0 legacy=0x7f884813d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:52.396 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.409+0000 7f8845d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:24:52.396 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.409+0000 7f8845d9b640 1 monclient: found mon.noname-b 2024-03-29T11:24:52.396 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.409+0000 7f8845d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:52.396 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.409+0000 7f8845d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.17:6790/0 2024-03-29T11:24:52.396 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.409+0000 7f8845d9b640 1 -- 172.21.15.17:0/3533401351 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88480accf0 con 0x7f88480aeca0 2024-03-29T11:24:52.396 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.409+0000 7f8845d9b640 10 monclient: _finish_auth 0 2024-03-29T11:24:52.396 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.409+0000 7f8845d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:24:52.396 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.409+0000 7f8845d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:52.396 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.409+0000 7f8845d9b640 1 -- 172.21.15.17:0/3533401351 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f8840001990 con 0x7f88480aeca0 2024-03-29T11:24:52.396 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.409+0000 7f8845d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:52.396 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.409+0000 7f8845d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:52.396 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.409+0000 7f8845d9b640 10 monclient: dump: 2024-03-29T11:24:52.396 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:24:52.396 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:52.396 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:52.397 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:52.397 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:52.397 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:52.397 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:24:52.397 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:24:52.397 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:24:52.397 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:52.397 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.409+0000 7f884d152640 5 monclient: authenticate success, global_id 4472 2024-03-29T11:24:52.397 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.409+0000 7f884d152640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:52.397 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.410+0000 7f8845d9b640 1 -- 172.21.15.17:0/3533401351 <== mon.2 v1:172.21.15.17:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8840001af0 con 0x7f88480aeca0 2024-03-29T11:24:52.397 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.410+0000 7f8845d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:52.397 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.410+0000 7f8845d9b640 1 -- 172.21.15.17:0/3533401351 <== mon.2 v1:172.21.15.17:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f8840001f40 con 0x7f88480aeca0 2024-03-29T11:24:52.397 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.410+0000 7f8845d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:52.397 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.410+0000 7f8845d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:52.397 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.410+0000 7f8845d9b640 10 monclient: dump: 2024-03-29T11:24:52.397 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:24:52.398 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:52.398 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:52.398 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:52.398 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:52.398 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:52.398 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:24:52.398 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:24:52.398 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:24:52.398 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:52.398 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.410+0000 7f884d152640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:52.398 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.410+0000 7f884d152640 4 set_mon_vals no callback set 2024-03-29T11:24:52.398 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.411+0000 7f884d152640 10 monclient: shutdown 2024-03-29T11:24:52.398 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.411+0000 7f884d152640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:52.398 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.411+0000 7f884d152640 1 -- 172.21.15.17:0/3533401351 >> v1:172.21.15.17:6790/0 conn(0x7f88480aeca0 legacy=0x7f884814b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:52.399 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.411+0000 7f884d152640 1 -- 172.21.15.17:0/3533401351 shutdown_connections 2024-03-29T11:24:52.399 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.411+0000 7f884d152640 1 -- 172.21.15.17:0/3533401351 >> 172.21.15.17:0/3533401351 conn(0x7f884806b660 msgr2=0x7f884806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:52.399 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.411+0000 7f884d152640 1 -- 172.21.15.17:0/3533401351 shutdown_connections 2024-03-29T11:24:52.400 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.411+0000 7f884d152640 1 -- 172.21.15.17:0/3533401351 wait complete. 2024-03-29T11:24:52.400 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.412+0000 7f884d152640 10 monclient: build_initial_monmap 2024-03-29T11:24:52.400 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.412+0000 7f884d152640 10 monclient: monmap: 2024-03-29T11:24:52.400 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:24:52.400 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:52.400 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:24:52.400 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:24:52.400 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:52.400 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:52.400 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:24:52.401 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:24:52.401 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:24:52.401 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:52.401 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.413+0000 7f884d152640 1 librados: starting msgr at 2024-03-29T11:24:52.401 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.413+0000 7f884d152640 1 librados: starting objecter 2024-03-29T11:24:52.401 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.413+0000 7f884d152640 1 Processor -- start 2024-03-29T11:24:52.401 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.413+0000 7f884d152640 1 -- start start 2024-03-29T11:24:52.401 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.413+0000 7f884d152640 1 librados: setting wanted keys 2024-03-29T11:24:52.401 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.413+0000 7f884d152640 1 librados: calling monclient init 2024-03-29T11:24:52.401 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.413+0000 7f884d152640 10 monclient: init 2024-03-29T11:24:52.401 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.414+0000 7f884d152640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:52.401 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.414+0000 7f884d152640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:24:52.401 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.414+0000 7f884d152640 10 monclient(hunting): picked mon.noname-a con 0x7f88480aeca0 addr v1:172.21.15.17:6789/0 2024-03-29T11:24:52.401 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.414+0000 7f884d152640 10 monclient(hunting): picked mon.noname-c con 0x7f88480b0500 addr v1:172.21.15.17:6790/0 2024-03-29T11:24:52.401 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.414+0000 7f884d152640 10 monclient(hunting): picked mon.noname-b con 0x7f88481d6860 addr v1:172.21.15.94:6789/0 2024-03-29T11:24:52.402 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.414+0000 7f884d152640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f884811d4e0 con 0x7f88480aeca0 2024-03-29T11:24:52.402 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.414+0000 7f884d152640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f884811eb90 con 0x7f88481d6860 2024-03-29T11:24:52.402 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.414+0000 7f884d152640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f884811c9f0 con 0x7f88480b0500 2024-03-29T11:24:52.402 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.414+0000 7f884d152640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:52.402 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.414+0000 7f884d152640 10 monclient(hunting): authenticate will time out at 1198.619639s 2024-03-29T11:24:52.402 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.415+0000 7f884659c640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f88480b0500 0x7f884813a9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:36634/0 (socket says 172.21.15.17:36634) 2024-03-29T11:24:52.402 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.415+0000 7f884659c640 1 -- 172.21.15.17:0/476479621 learned_addr learned my addr 172.21.15.17:0/476479621 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:24:52.402 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.416+0000 7f883f7fe640 10 client.?.objecter ms_handle_connect 0x7f88480b0500 2024-03-29T11:24:52.402 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.416+0000 7f883f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:52.402 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.416+0000 7f883f7fe640 1 -- 172.21.15.17:0/476479621 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1227710619 0 0) 0x7f884811c9f0 con 0x7f88480b0500 2024-03-29T11:24:52.402 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.416+0000 7f883f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:52.402 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.416+0000 7f883f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:52.402 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.416+0000 7f883f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:52.403 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.416+0000 7f883f7fe640 10 monclient(hunting): my global_id is 4475 2024-03-29T11:24:52.403 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.416+0000 7f883f7fe640 1 -- 172.21.15.17:0/476479621 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8828004eb0 con 0x7f88480b0500 2024-03-29T11:24:52.407 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.418+0000 7f883f7fe640 1 -- 172.21.15.17:0/476479621 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 983088068 0 0) 0x7f8828004eb0 con 0x7f88480b0500 2024-03-29T11:24:52.407 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.419+0000 7f883f7fe640 1 -- 172.21.15.17:0/476479621 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f884811c9f0 con 0x7f88480b0500 2024-03-29T11:24:52.407 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.419+0000 7f883f7fe640 10 client.?.objecter ms_handle_connect 0x7f88480aeca0 2024-03-29T11:24:52.407 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.419+0000 7f883f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:52.407 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.419+0000 7f883f7fe640 1 -- 172.21.15.17:0/476479621 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f8830001860 con 0x7f88480b0500 2024-03-29T11:24:52.408 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.419+0000 7f883f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:52.408 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.419+0000 7f883f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:52.408 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.419+0000 7f883f7fe640 10 monclient(hunting): dump: 2024-03-29T11:24:52.408 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:24:52.408 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:52.408 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:52.408 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:52.408 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:52.408 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:52.408 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:24:52.408 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:24:52.408 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:24:52.408 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:52.408 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.419+0000 7f883f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:52.409 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.419+0000 7f883f7fe640 1 -- 172.21.15.17:0/476479621 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4294671036 0 0) 0x7f884811d4e0 con 0x7f88480aeca0 2024-03-29T11:24:52.409 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.419+0000 7f883f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:52.409 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.419+0000 7f883f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:52.409 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.419+0000 7f883f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:52.409 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.419+0000 7f883f7fe640 10 monclient(hunting): my global_id is 4428 2024-03-29T11:24:52.409 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.420+0000 7f883f7fe640 1 -- 172.21.15.17:0/476479621 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8828004eb0 con 0x7f88480aeca0 2024-03-29T11:24:52.409 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.420+0000 7f883f7fe640 1 -- 172.21.15.17:0/476479621 <== mon.2 v1:172.21.15.17:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3023594428 0 0) 0x7f884811c9f0 con 0x7f88480b0500 2024-03-29T11:24:52.409 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.420+0000 7f883f7fe640 1 -- 172.21.15.17:0/476479621 >> v1:172.21.15.94:6789/0 conn(0x7f88481d6860 legacy=0x7f88481de4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:24:52.409 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.420+0000 7f883f7fe640 1 -- 172.21.15.17:0/476479621 >> v1:172.21.15.17:6789/0 conn(0x7f88480aeca0 legacy=0x7f88481d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:52.409 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.420+0000 7f883f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:24:52.409 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.420+0000 7f883f7fe640 1 monclient: found mon.c 2024-03-29T11:24:52.409 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.420+0000 7f883f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:52.409 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.420+0000 7f883f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:24:52.409 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.420+0000 7f883f7fe640 1 -- 172.21.15.17:0/476479621 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f884811d970 con 0x7f88480b0500 2024-03-29T11:24:52.409 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.420+0000 7f883f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:24:52.409 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.420+0000 7f883f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:24:52.410 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.420+0000 7f883f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:52.410 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.420+0000 7f884d152640 5 monclient: authenticate success, global_id 4475 2024-03-29T11:24:52.410 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.420+0000 7f884d152640 10 monclient: _renew_subs 2024-03-29T11:24:52.410 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.420+0000 7f884d152640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:24:52.410 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.420+0000 7f884d152640 1 -- 172.21.15.17:0/476479621 --> v1:172.21.15.17:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f884813b0f0 con 0x7f88480b0500 2024-03-29T11:24:52.410 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.420+0000 7f884d152640 10 client.4475.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:52.410 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.420+0000 7f884d152640 10 monclient: _renew_subs 2024-03-29T11:24:52.410 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.420+0000 7f884d152640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:24:52.410 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.420+0000 7f884d152640 1 -- 172.21.15.17:0/476479621 --> v1:172.21.15.17:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f884806fe80 con 0x7f88480b0500 2024-03-29T11:24:52.410 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.420+0000 7f884d152640 1 librados: init done 2024-03-29T11:24:52.410 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.421+0000 7f883f7fe640 1 -- 172.21.15.17:0/476479621 <== mon.2 v1:172.21.15.17:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8830001a30 con 0x7f88480b0500 2024-03-29T11:24:52.410 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.421+0000 7f883f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:52.410 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.421+0000 7f884559a640 4 set_mon_vals no callback set 2024-03-29T11:24:52.411 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.421+0000 7f883f7fe640 1 -- 172.21.15.17:0/476479621 <== mon.2 v1:172.21.15.17:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f88300037f0 con 0x7f88480b0500 2024-03-29T11:24:52.411 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.421+0000 7f883f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:52.411 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.421+0000 7f883f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:52.411 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.422+0000 7f883f7fe640 10 monclient: dump: 2024-03-29T11:24:52.411 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:24:52.411 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:52.411 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:52.411 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:52.411 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:52.411 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:52.411 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:24:52.411 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:24:52.411 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:24:52.412 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:52.412 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.422+0000 7f884d152640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:52.412 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.423+0000 7f884d152640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:52.412 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.423+0000 7f884d152640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:24:52.412 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.423+0000 7f883f7fe640 1 -- 172.21.15.17:0/476479621 <== mon.2 v1:172.21.15.17:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f88300039d0 con 0x7f88480b0500 2024-03-29T11:24:52.412 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.423+0000 7f884d152640 1 -- 172.21.15.17:0/476479621 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f882c005080 con 0x7f88480b0500 2024-03-29T11:24:52.412 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.423+0000 7f883f7fe640 1 -- 172.21.15.17:0/476479621 <== mon.2 v1:172.21.15.17:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f88300959f0 con 0x7f88480b0500 2024-03-29T11:24:52.412 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.424+0000 7f883f7fe640 10 client.4475.objecter ms_dispatch 0x7f8848000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:52.412 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.424+0000 7f883f7fe640 3 client.4475.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:52.412 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.424+0000 7f8846d9d640 10 monclient: get_auth_request con 0x7f882806dcb0 auth_method 0 2024-03-29T11:24:52.412 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.424+0000 7f883f7fe640 3 client.4475.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:52.412 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.424+0000 7f883f7fe640 20 client.4475.objecter dump_active .. 0 homeless 2024-03-29T11:24:52.412 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.424+0000 7f883f7fe640 10 client.4475.objecter ms_handle_connect 0x7f882806dcb0 2024-03-29T11:24:52.415 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.432+0000 7f883f7fe640 1 -- 172.21.15.17:0/476479621 <== mon.2 v1:172.21.15.17:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f88300612d0 con 0x7f88480b0500 2024-03-29T11:24:52.415 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.432+0000 7f883f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:52.415 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.432+0000 7f883f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:52.748 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.765+0000 7f884d152640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:52.748 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.765+0000 7f884d152640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:52.748 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.765+0000 7f884d152640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:24:52.748 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.765+0000 7f884d152640 1 -- 172.21.15.17:0/476479621 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f882c005330 con 0x7f88480b0500 2024-03-29T11:24:52.749 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.767+0000 7f883f7fe640 1 -- 172.21.15.17:0/476479621 <== mon.2 v1:172.21.15.17:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f8830063200 con 0x7f88480b0500 2024-03-29T11:24:52.750 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.767+0000 7f883f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:52.750 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.767+0000 7f883f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:52.750 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:24:52.750 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:52.754 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.771+0000 7f884d152640 10 librados: watch_flush enter 2024-03-29T11:24:52.755 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.771+0000 7f884d152640 10 librados: watch_flush exit 2024-03-29T11:24:52.755 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.771+0000 7f884d152640 20 client.4475.objecter shutdown clearing up homeless session... 2024-03-29T11:24:52.755 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.771+0000 7f884d152640 10 client.4475.objecter successfully canceled tick 2024-03-29T11:24:52.755 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.771+0000 7f884d152640 1 -- 172.21.15.17:0/476479621 >> v1:172.21.15.17:6816/3236185052 conn(0x7f882806dcb0 legacy=0x7f8828070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:52.755 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.771+0000 7f884d152640 10 monclient: shutdown 2024-03-29T11:24:52.755 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.771+0000 7f884d152640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:52.755 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.771+0000 7f884d152640 1 -- 172.21.15.17:0/476479621 >> v1:172.21.15.17:6790/0 conn(0x7f88480b0500 legacy=0x7f884813a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:52.755 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.771+0000 7f884d152640 1 -- 172.21.15.17:0/476479621 shutdown_connections 2024-03-29T11:24:52.755 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.771+0000 7f884d152640 1 -- 172.21.15.17:0/476479621 >> 172.21.15.17:0/476479621 conn(0x7f884806b660 msgr2=0x7f884806ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:52.755 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.771+0000 7f884d152640 1 -- 172.21.15.17:0/476479621 shutdown_connections 2024-03-29T11:24:52.755 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.771+0000 7f884d152640 1 -- 172.21.15.17:0/476479621 wait complete. 2024-03-29T11:24:52.755 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:52.771+0000 7f884d152640 1 librados: shutdown 2024-03-29T11:24:52.770 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:59.577 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:59.733 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.750+0000 7f8f1a1c4640 10 monclient: get_monmap_and_config 2024-03-29T11:24:59.733 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.750+0000 7f8f1a1c4640 10 monclient: build_initial_monmap 2024-03-29T11:24:59.733 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.750+0000 7f8f1a1c4640 10 monclient: monmap: 2024-03-29T11:24:59.733 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:24:59.734 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:59.734 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:24:59.751547+0000 2024-03-29T11:24:59.734 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:24:59.751547+0000 2024-03-29T11:24:59.734 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:59.734 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:59.734 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:24:59.734 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:24:59.734 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:24:59.734 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:59.735 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.751+0000 7f8f1a1c4640 1 Processor -- start 2024-03-29T11:24:59.735 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.751+0000 7f8f1a1c4640 1 -- start start 2024-03-29T11:24:59.735 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.751+0000 7f8f1a1c4640 10 monclient: init 2024-03-29T11:24:59.737 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.752+0000 7f8f1a1c4640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:59.738 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.752+0000 7f8f1a1c4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:24:59.746 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.752+0000 7f8f1a1c4640 10 monclient(hunting): picked mon.noname-a con 0x7f8f141457f0 addr v1:172.21.15.17:6789/0 2024-03-29T11:24:59.746 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.755+0000 7f8f1a1c4640 10 monclient(hunting): picked mon.noname-b con 0x7f8f1413f650 addr v1:172.21.15.17:6790/0 2024-03-29T11:24:59.746 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.758+0000 7f8f1a1c4640 10 monclient(hunting): picked mon.noname-c con 0x7f8f140a9d70 addr v1:172.21.15.94:6789/0 2024-03-29T11:24:59.746 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.758+0000 7f8f1a1c4640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f140aaef0 con 0x7f8f141457f0 2024-03-29T11:24:59.746 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.758+0000 7f8f137fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f8f141457f0 0x7f8f1413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:58369/0 (socket says 172.21.15.17:58369) 2024-03-29T11:24:59.746 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.758+0000 7f8f137fe640 1 -- 172.21.15.17:0/1390958624 learned_addr learned my addr 172.21.15.17:0/1390958624 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:24:59.746 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.758+0000 7f8f1a1c4640 1 -- 172.21.15.17:0/1390958624 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f14145bd0 con 0x7f8f140a9d70 2024-03-29T11:24:59.746 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.758+0000 7f8f1a1c4640 1 -- 172.21.15.17:0/1390958624 --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f14145eb0 con 0x7f8f1413f650 2024-03-29T11:24:59.747 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.758+0000 7f8f1a1c4640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:59.747 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.758+0000 7f8f1a1c4640 10 monclient(hunting): authenticate will time out at 1205.963340s 2024-03-29T11:24:59.747 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.758+0000 7f8f127fc640 1 -- 172.21.15.17:0/1390958624 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3433342045 0 0) 0x7f8f14145eb0 con 0x7f8f1413f650 2024-03-29T11:24:59.747 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.759+0000 7f8f127fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:59.747 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.759+0000 7f8f127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:59.747 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.759+0000 7f8f127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:59.747 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.759+0000 7f8f127fc640 10 monclient(hunting): my global_id is 4478 2024-03-29T11:24:59.747 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.759+0000 7f8f127fc640 1 -- 172.21.15.17:0/1390958624 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8efc003520 con 0x7f8f1413f650 2024-03-29T11:24:59.747 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.759+0000 7f8f127fc640 1 -- 172.21.15.17:0/1390958624 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3207400538 0 0) 0x7f8f14145bd0 con 0x7f8f140a9d70 2024-03-29T11:24:59.747 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.759+0000 7f8f127fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:59.747 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.759+0000 7f8f127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:59.747 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.759+0000 7f8f127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:59.747 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.759+0000 7f8f127fc640 10 monclient(hunting): my global_id is 4462 2024-03-29T11:24:59.747 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.759+0000 7f8f127fc640 1 -- 172.21.15.17:0/1390958624 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f14145eb0 con 0x7f8f140a9d70 2024-03-29T11:24:59.747 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.759+0000 7f8f127fc640 1 -- 172.21.15.17:0/1390958624 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2183036627 0 0) 0x7f8f140aaef0 con 0x7f8f141457f0 2024-03-29T11:24:59.747 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.759+0000 7f8f127fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:59.748 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.759+0000 7f8f127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:59.748 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.759+0000 7f8f127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:59.748 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.759+0000 7f8f127fc640 10 monclient(hunting): my global_id is 4431 2024-03-29T11:24:59.748 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.759+0000 7f8f127fc640 1 -- 172.21.15.17:0/1390958624 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f14145bd0 con 0x7f8f141457f0 2024-03-29T11:24:59.748 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.759+0000 7f8f127fc640 1 -- 172.21.15.17:0/1390958624 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3378408007 0 0) 0x7f8efc003520 con 0x7f8f1413f650 2024-03-29T11:24:59.748 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.759+0000 7f8f127fc640 1 -- 172.21.15.17:0/1390958624 >> v1:172.21.15.94:6789/0 conn(0x7f8f140a9d70 legacy=0x7f8f1414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:59.748 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.759+0000 7f8f127fc640 1 -- 172.21.15.17:0/1390958624 >> v1:172.21.15.17:6789/0 conn(0x7f8f141457f0 legacy=0x7f8f1413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:59.748 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.759+0000 7f8f127fc640 10 monclient: _finish_hunting 0 2024-03-29T11:24:59.748 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.759+0000 7f8f127fc640 1 monclient: found mon.noname-b 2024-03-29T11:24:59.748 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.759+0000 7f8f127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:59.748 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.759+0000 7f8f127fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.17:6790/0 2024-03-29T11:24:59.748 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.759+0000 7f8f127fc640 1 -- 172.21.15.17:0/1390958624 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f14146340 con 0x7f8f1413f650 2024-03-29T11:24:59.748 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.759+0000 7f8f127fc640 10 monclient: _finish_auth 0 2024-03-29T11:24:59.748 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.759+0000 7f8f127fc640 10 monclient: _check_auth_tickets 2024-03-29T11:24:59.748 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.759+0000 7f8f127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:59.749 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.759+0000 7f8f127fc640 1 -- 172.21.15.17:0/1390958624 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f8f08001690 con 0x7f8f1413f650 2024-03-29T11:24:59.749 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.759+0000 7f8f127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:59.749 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.759+0000 7f8f127fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:59.749 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.759+0000 7f8f127fc640 10 monclient: dump: 2024-03-29T11:24:59.749 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:24:59.749 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:59.749 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:59.749 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:59.749 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:59.749 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:59.749 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:24:59.749 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:24:59.749 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:24:59.749 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:59.750 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.759+0000 7f8f1a1c4640 5 monclient: authenticate success, global_id 4478 2024-03-29T11:24:59.750 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.759+0000 7f8f1a1c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:59.750 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.760+0000 7f8f127fc640 1 -- 172.21.15.17:0/1390958624 <== mon.2 v1:172.21.15.17:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f080017f0 con 0x7f8f1413f650 2024-03-29T11:24:59.750 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.760+0000 7f8f127fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:59.750 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.760+0000 7f8f1a1c4640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:59.750 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.760+0000 7f8f1a1c4640 4 set_mon_vals no callback set 2024-03-29T11:24:59.750 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.760+0000 7f8f127fc640 1 -- 172.21.15.17:0/1390958624 <== mon.2 v1:172.21.15.17:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f8f08001c00 con 0x7f8f1413f650 2024-03-29T11:24:59.750 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.760+0000 7f8f127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:59.750 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.760+0000 7f8f127fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:59.750 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.760+0000 7f8f127fc640 10 monclient: dump: 2024-03-29T11:24:59.750 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:24:59.750 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:59.750 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:59.750 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:59.751 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:59.751 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:59.751 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:24:59.751 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:24:59.751 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:24:59.751 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:59.751 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.761+0000 7f8f1a1c4640 10 monclient: shutdown 2024-03-29T11:24:59.751 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.761+0000 7f8f1a1c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:59.751 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.761+0000 7f8f1a1c4640 1 -- 172.21.15.17:0/1390958624 >> v1:172.21.15.17:6790/0 conn(0x7f8f1413f650 legacy=0x7f8f140b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:59.751 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.762+0000 7f8f1a1c4640 1 -- 172.21.15.17:0/1390958624 shutdown_connections 2024-03-29T11:24:59.751 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.762+0000 7f8f1a1c4640 1 -- 172.21.15.17:0/1390958624 >> 172.21.15.17:0/1390958624 conn(0x7f8f14066a00 msgr2=0x7f8f14066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:59.751 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.762+0000 7f8f1a1c4640 1 -- 172.21.15.17:0/1390958624 shutdown_connections 2024-03-29T11:24:59.761 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.762+0000 7f8f1a1c4640 1 -- 172.21.15.17:0/1390958624 wait complete. 2024-03-29T11:24:59.761 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.763+0000 7f8f1a1c4640 10 monclient: build_initial_monmap 2024-03-29T11:24:59.761 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.763+0000 7f8f1a1c4640 10 monclient: monmap: 2024-03-29T11:24:59.761 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:24:59.762 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:59.762 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:24:59.762 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:24:59.762 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:59.762 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:59.762 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:24:59.762 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:24:59.762 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:24:59.762 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:59.762 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.764+0000 7f8f1a1c4640 1 librados: starting msgr at 2024-03-29T11:24:59.762 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.764+0000 7f8f1a1c4640 1 librados: starting objecter 2024-03-29T11:24:59.762 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.764+0000 7f8f1a1c4640 1 Processor -- start 2024-03-29T11:24:59.762 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.764+0000 7f8f1a1c4640 1 -- start start 2024-03-29T11:24:59.763 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.764+0000 7f8f1a1c4640 1 librados: setting wanted keys 2024-03-29T11:24:59.763 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.764+0000 7f8f1a1c4640 1 librados: calling monclient init 2024-03-29T11:24:59.763 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.764+0000 7f8f1a1c4640 10 monclient: init 2024-03-29T11:24:59.763 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.765+0000 7f8f1a1c4640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:59.763 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.765+0000 7f8f1a1c4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:24:59.763 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.768+0000 7f8f1a1c4640 10 monclient(hunting): picked mon.noname-c con 0x7f8f140a9d70 addr v1:172.21.15.17:6790/0 2024-03-29T11:24:59.763 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.768+0000 7f8f137fe640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f8f140a9d70 0x7f8f14152c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:58277/0 (socket says 172.21.15.17:58277) 2024-03-29T11:24:59.763 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.768+0000 7f8f137fe640 1 -- 172.21.15.17:0/3680460874 learned_addr learned my addr 172.21.15.17:0/3680460874 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:24:59.763 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.769+0000 7f8f1a1c4640 10 monclient(hunting): picked mon.noname-b con 0x7f8f1413f650 addr v1:172.21.15.94:6789/0 2024-03-29T11:24:59.763 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.769+0000 7f8f1a1c4640 10 monclient(hunting): picked mon.noname-a con 0x7f8f141457f0 addr v1:172.21.15.17:6789/0 2024-03-29T11:24:59.763 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.769+0000 7f8f1a1c4640 1 -- 172.21.15.17:0/3680460874 --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f140ab1d0 con 0x7f8f141457f0 2024-03-29T11:24:59.763 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.769+0000 7f8f1a1c4640 1 -- 172.21.15.17:0/3680460874 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f1406fc70 con 0x7f8f1413f650 2024-03-29T11:24:59.763 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.769+0000 7f8f1a1c4640 1 -- 172.21.15.17:0/3680460874 --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f1406fec0 con 0x7f8f140a9d70 2024-03-29T11:24:59.764 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.769+0000 7f8f1a1c4640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:59.764 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.769+0000 7f8f1a1c4640 10 monclient(hunting): authenticate will time out at 1205.974639s 2024-03-29T11:24:59.764 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.770+0000 7f8efbfff640 10 client.?.objecter ms_handle_connect 0x7f8f140a9d70 2024-03-29T11:24:59.764 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.770+0000 7f8efbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:59.764 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.772+0000 7f8efbfff640 10 client.?.objecter ms_handle_connect 0x7f8f1413f650 2024-03-29T11:24:59.764 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.773+0000 7f8efbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:59.764 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.774+0000 7f8efbfff640 10 client.?.objecter ms_handle_connect 0x7f8f141457f0 2024-03-29T11:24:59.764 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.774+0000 7f8efbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:59.764 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.774+0000 7f8efbfff640 1 -- 172.21.15.17:0/3680460874 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 603807875 0 0) 0x7f8f1406fc70 con 0x7f8f1413f650 2024-03-29T11:24:59.765 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.774+0000 7f8efbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:59.765 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.774+0000 7f8efbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:59.765 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.774+0000 7f8efbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:59.765 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.774+0000 7f8efbfff640 10 monclient(hunting): my global_id is 4465 2024-03-29T11:24:59.765 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.774+0000 7f8efbfff640 1 -- 172.21.15.17:0/3680460874 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f00004eb0 con 0x7f8f1413f650 2024-03-29T11:24:59.765 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.774+0000 7f8efbfff640 1 -- 172.21.15.17:0/3680460874 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1750475559 0 0) 0x7f8f140ab1d0 con 0x7f8f141457f0 2024-03-29T11:24:59.765 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.774+0000 7f8efbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:59.765 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.774+0000 7f8efbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:59.765 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.774+0000 7f8efbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:59.765 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.774+0000 7f8efbfff640 10 monclient(hunting): my global_id is 4434 2024-03-29T11:24:59.765 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.774+0000 7f8efbfff640 1 -- 172.21.15.17:0/3680460874 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f1406fc70 con 0x7f8f141457f0 2024-03-29T11:24:59.765 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.774+0000 7f8efbfff640 1 -- 172.21.15.17:0/3680460874 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 11276949 0 0) 0x7f8f1406fec0 con 0x7f8f140a9d70 2024-03-29T11:24:59.765 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.774+0000 7f8efbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:59.765 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.774+0000 7f8efbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:59.765 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.774+0000 7f8efbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:59.766 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.774+0000 7f8efbfff640 10 monclient(hunting): my global_id is 4481 2024-03-29T11:24:59.766 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.774+0000 7f8efbfff640 1 -- 172.21.15.17:0/3680460874 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f140ab1d0 con 0x7f8f140a9d70 2024-03-29T11:24:59.766 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.775+0000 7f8efbfff640 1 -- 172.21.15.17:0/3680460874 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3121027138 0 0) 0x7f8f1406fc70 con 0x7f8f141457f0 2024-03-29T11:24:59.766 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.775+0000 7f8efbfff640 1 -- 172.21.15.17:0/3680460874 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f1406fec0 con 0x7f8f141457f0 2024-03-29T11:24:59.766 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.775+0000 7f8efbfff640 1 -- 172.21.15.17:0/3680460874 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1179696811 0 0) 0x7f8f140ab1d0 con 0x7f8f140a9d70 2024-03-29T11:24:59.766 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.775+0000 7f8efbfff640 1 -- 172.21.15.17:0/3680460874 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f1406fc70 con 0x7f8f140a9d70 2024-03-29T11:24:59.766 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.775+0000 7f8efbfff640 1 -- 172.21.15.17:0/3680460874 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 487267106 0 0) 0x7f8f00004eb0 con 0x7f8f1413f650 2024-03-29T11:24:59.766 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.775+0000 7f8efbfff640 1 -- 172.21.15.17:0/3680460874 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f140ab1d0 con 0x7f8f1413f650 2024-03-29T11:24:59.766 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.775+0000 7f8efbfff640 1 -- 172.21.15.17:0/3680460874 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f8f0c001b50 con 0x7f8f141457f0 2024-03-29T11:24:59.766 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.775+0000 7f8efbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:59.766 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.775+0000 7f8efbfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:59.766 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.775+0000 7f8efbfff640 10 monclient(hunting): dump: 2024-03-29T11:24:59.766 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:24:59.766 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:59.767 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:59.767 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:59.767 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:59.767 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:59.767 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:24:59.767 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:24:59.767 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:24:59.767 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:59.767 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.775+0000 7f8efbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:59.767 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.775+0000 7f8efbfff640 1 -- 172.21.15.17:0/3680460874 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f8f04001870 con 0x7f8f140a9d70 2024-03-29T11:24:59.767 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.775+0000 7f8efbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:59.767 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.775+0000 7f8efbfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:59.767 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.775+0000 7f8efbfff640 10 monclient(hunting): dump: 2024-03-29T11:24:59.767 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:24:59.767 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:59.768 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:59.768 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:59.768 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:59.768 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:59.768 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:24:59.768 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:24:59.768 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:24:59.768 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:59.768 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.775+0000 7f8efbfff640 1 -- 172.21.15.17:0/3680460874 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f8f08001b00 con 0x7f8f1413f650 2024-03-29T11:24:59.768 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.775+0000 7f8efbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:59.768 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.775+0000 7f8efbfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:59.768 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.775+0000 7f8efbfff640 10 monclient(hunting): dump: 2024-03-29T11:24:59.769 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:24:59.769 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:59.769 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:59.769 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:59.769 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:59.769 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:59.769 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:24:59.769 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:24:59.769 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:24:59.769 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:59.769 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.776+0000 7f8efbfff640 1 -- 172.21.15.17:0/3680460874 <== mon.1 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4028337340 0 0) 0x7f8f140ab1d0 con 0x7f8f1413f650 2024-03-29T11:24:59.769 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.776+0000 7f8efbfff640 1 -- 172.21.15.17:0/3680460874 >> v1:172.21.15.17:6790/0 conn(0x7f8f140a9d70 legacy=0x7f8f14152c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:59.770 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.776+0000 7f8efbfff640 1 -- 172.21.15.17:0/3680460874 >> v1:172.21.15.17:6789/0 conn(0x7f8f141457f0 legacy=0x7f8f140712f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:59.770 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.776+0000 7f8efbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:24:59.770 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.776+0000 7f8efbfff640 1 monclient: found mon.b 2024-03-29T11:24:59.770 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.776+0000 7f8efbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:59.770 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.776+0000 7f8efbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:24:59.770 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.776+0000 7f8efbfff640 1 -- 172.21.15.17:0/3680460874 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f140701a0 con 0x7f8f1413f650 2024-03-29T11:24:59.770 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.776+0000 7f8efbfff640 10 monclient: _finish_auth 0 2024-03-29T11:24:59.770 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.776+0000 7f8efbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:24:59.770 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.776+0000 7f8efbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:59.770 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.776+0000 7f8f1a1c4640 5 monclient: authenticate success, global_id 4465 2024-03-29T11:24:59.770 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.776+0000 7f8f1a1c4640 10 monclient: _renew_subs 2024-03-29T11:24:59.770 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.776+0000 7f8f1a1c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:24:59.770 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.776+0000 7f8f1a1c4640 1 -- 172.21.15.17:0/3680460874 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f14070480 con 0x7f8f1413f650 2024-03-29T11:24:59.771 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.776+0000 7f8f1a1c4640 10 client.4465.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:59.771 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.776+0000 7f8f1a1c4640 10 monclient: _renew_subs 2024-03-29T11:24:59.771 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.776+0000 7f8f1a1c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:24:59.771 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.776+0000 7f8f1a1c4640 1 -- 172.21.15.17:0/3680460874 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f1406f1f0 con 0x7f8f1413f650 2024-03-29T11:24:59.771 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.776+0000 7f8f1a1c4640 1 librados: init done 2024-03-29T11:24:59.771 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.778+0000 7f8f1a1c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:59.771 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.778+0000 7f8f1a1c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:59.771 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.778+0000 7f8f1a1c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:24:59.771 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.778+0000 7f8f1a1c4640 1 -- 172.21.15.17:0/3680460874 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f140708a0 con 0x7f8f1413f650 2024-03-29T11:24:59.771 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.778+0000 7f8efbfff640 1 -- 172.21.15.17:0/3680460874 <== mon.1 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f08001c60 con 0x7f8f1413f650 2024-03-29T11:24:59.771 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.778+0000 7f8efbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:59.771 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.778+0000 7f8efbfff640 1 -- 172.21.15.17:0/3680460874 <== mon.1 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f8f08003a40 con 0x7f8f1413f650 2024-03-29T11:24:59.772 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.778+0000 7f8efbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:59.772 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.778+0000 7f8efbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:59.772 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.778+0000 7f8efbfff640 10 monclient: dump: 2024-03-29T11:24:59.772 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:24:59.772 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:24:59.772 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:59.772 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:24:59.772 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:59.772 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:24:59.772 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:24:59.772 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:24:59.772 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:24:59.772 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:24:59.774 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.779+0000 7f8f117fa640 4 set_mon_vals no callback set 2024-03-29T11:24:59.774 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.783+0000 7f8efbfff640 1 -- 172.21.15.17:0/3680460874 <== mon.1 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f8f08019a10 con 0x7f8f1413f650 2024-03-29T11:24:59.775 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.784+0000 7f8efbfff640 1 -- 172.21.15.17:0/3680460874 <== mon.1 v1:172.21.15.94:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f8f080837e0 con 0x7f8f1413f650 2024-03-29T11:24:59.775 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.784+0000 7f8f137fe640 10 monclient: get_auth_request con 0x7f8f0006e100 auth_method 0 2024-03-29T11:24:59.775 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.784+0000 7f8efbfff640 10 client.4465.objecter ms_dispatch 0x7f8f14000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:59.775 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.784+0000 7f8efbfff640 3 client.4465.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:59.775 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.784+0000 7f8efbfff640 3 client.4465.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:59.775 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.784+0000 7f8efbfff640 20 client.4465.objecter dump_active .. 0 homeless 2024-03-29T11:24:59.775 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.784+0000 7f8efbfff640 10 client.4465.objecter ms_handle_connect 0x7f8f0006e100 2024-03-29T11:24:59.775 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.787+0000 7f8efbfff640 1 -- 172.21.15.17:0/3680460874 <== mon.1 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8f0804f0c0 con 0x7f8f1413f650 2024-03-29T11:24:59.775 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.787+0000 7f8efbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:59.776 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:24:59.787+0000 7f8efbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:00.103 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:00.119+0000 7f8f1a1c4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:00.103 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:00.119+0000 7f8f1a1c4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:00.103 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:00.119+0000 7f8f1a1c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:25:00.103 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:00.119+0000 7f8f1a1c4640 1 -- 172.21.15.17:0/3680460874 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8f14075f10 con 0x7f8f1413f650 2024-03-29T11:25:00.104 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:00.121+0000 7f8efbfff640 1 -- 172.21.15.17:0/3680460874 <== mon.1 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f8f08050ff0 con 0x7f8f1413f650 2024-03-29T11:25:00.104 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:00.121+0000 7f8efbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:00.104 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:00.121+0000 7f8efbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:00.104 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:25:00.104 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:00.106 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:00.124+0000 7f8f1a1c4640 10 librados: watch_flush enter 2024-03-29T11:25:00.109 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:00.126+0000 7f8f1a1c4640 10 librados: watch_flush exit 2024-03-29T11:25:00.109 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:00.126+0000 7f8f1a1c4640 20 client.4465.objecter shutdown clearing up homeless session... 2024-03-29T11:25:00.109 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:00.126+0000 7f8f1a1c4640 10 client.4465.objecter successfully canceled tick 2024-03-29T11:25:00.109 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:00.126+0000 7f8f1a1c4640 1 -- 172.21.15.17:0/3680460874 >> v1:172.21.15.17:6816/3236185052 conn(0x7f8f0006e100 legacy=0x7f8f000705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:00.109 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:00.126+0000 7f8f1a1c4640 10 monclient: shutdown 2024-03-29T11:25:00.109 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:00.126+0000 7f8f1a1c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:00.109 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:00.126+0000 7f8f1a1c4640 1 -- 172.21.15.17:0/3680460874 >> v1:172.21.15.94:6789/0 conn(0x7f8f1413f650 legacy=0x7f8f14070be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:00.110 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:00.127+0000 7f8f1a1c4640 1 -- 172.21.15.17:0/3680460874 shutdown_connections 2024-03-29T11:25:00.110 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:00.127+0000 7f8f1a1c4640 1 -- 172.21.15.17:0/3680460874 >> 172.21.15.17:0/3680460874 conn(0x7f8f14066a00 msgr2=0x7f8f1413dee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:00.110 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:00.127+0000 7f8f1a1c4640 1 -- 172.21.15.17:0/3680460874 shutdown_connections 2024-03-29T11:25:00.110 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:00.127+0000 7f8f1a1c4640 1 -- 172.21.15.17:0/3680460874 wait complete. 2024-03-29T11:25:00.111 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:00.127+0000 7f8f1a1c4640 1 librados: shutdown 2024-03-29T11:25:00.128 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:06.935 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:07.086 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.102+0000 7fdfe6006640 10 monclient: get_monmap_and_config 2024-03-29T11:25:07.086 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.102+0000 7fdfe6006640 10 monclient: build_initial_monmap 2024-03-29T11:25:07.086 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.102+0000 7fdfe6006640 10 monclient: monmap: 2024-03-29T11:25:07.086 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:25:07.086 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:25:07.086 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:25:07.103484+0000 2024-03-29T11:25:07.086 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:25:07.103484+0000 2024-03-29T11:25:07.086 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:07.086 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:07.087 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:25:07.087 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:25:07.087 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:25:07.087 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:07.088 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.103+0000 7fdfe6006640 1 Processor -- start 2024-03-29T11:25:07.088 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.103+0000 7fdfe6006640 1 -- start start 2024-03-29T11:25:07.088 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.103+0000 7fdfe6006640 10 monclient: init 2024-03-29T11:25:07.088 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.104+0000 7fdfe6006640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:07.088 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.104+0000 7fdfe6006640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:25:07.088 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.104+0000 7fdfe6006640 10 monclient(hunting): picked mon.noname-c con 0x7fdfe0071de0 addr v1:172.21.15.94:6789/0 2024-03-29T11:25:07.088 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.104+0000 7fdfe6006640 10 monclient(hunting): picked mon.noname-b con 0x7fdfe00721c0 addr v1:172.21.15.17:6790/0 2024-03-29T11:25:07.089 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.105+0000 7fdfe6006640 10 monclient(hunting): picked mon.noname-a con 0x7fdfe0145ae0 addr v1:172.21.15.17:6789/0 2024-03-29T11:25:07.089 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.105+0000 7fdfe6006640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfe0145650 con 0x7fdfe0145ae0 2024-03-29T11:25:07.089 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.105+0000 7fdfe6006640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfe00ae260 con 0x7fdfe0071de0 2024-03-29T11:25:07.089 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.105+0000 7fdfe6006640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfe01420d0 con 0x7fdfe00721c0 2024-03-29T11:25:07.089 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.105+0000 7fdfe6006640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:07.089 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.105+0000 7fdfe6006640 10 monclient(hunting): authenticate will time out at 1213.310379s 2024-03-29T11:25:07.092 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.108+0000 7fdfdf7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fdfe0071de0 0x7fdfe00aed80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:58529/0 (socket says 172.21.15.17:58529) 2024-03-29T11:25:07.092 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.108+0000 7fdfdf7fe640 1 -- 172.21.15.17:0/2120079874 learned_addr learned my addr 172.21.15.17:0/2120079874 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:25:07.096 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.111+0000 7fdfde7fc640 1 -- 172.21.15.17:0/2120079874 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1793357901 0 0) 0x7fdfe01420d0 con 0x7fdfe00721c0 2024-03-29T11:25:07.096 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.111+0000 7fdfde7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:07.096 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.111+0000 7fdfde7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:07.096 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.111+0000 7fdfde7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:07.096 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.111+0000 7fdfde7fc640 10 monclient(hunting): my global_id is 4484 2024-03-29T11:25:07.096 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.111+0000 7fdfde7fc640 1 -- 172.21.15.17:0/2120079874 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfc8003520 con 0x7fdfe00721c0 2024-03-29T11:25:07.096 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.112+0000 7fdfde7fc640 1 -- 172.21.15.17:0/2120079874 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1729275108 0 0) 0x7fdfe00ae260 con 0x7fdfe0071de0 2024-03-29T11:25:07.096 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.112+0000 7fdfde7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:07.096 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.112+0000 7fdfde7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:07.096 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.112+0000 7fdfde7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:07.096 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.112+0000 7fdfde7fc640 10 monclient(hunting): my global_id is 4468 2024-03-29T11:25:07.096 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.112+0000 7fdfde7fc640 1 -- 172.21.15.17:0/2120079874 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfe01420d0 con 0x7fdfe0071de0 2024-03-29T11:25:07.096 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.112+0000 7fdfde7fc640 1 -- 172.21.15.17:0/2120079874 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3566121898 0 0) 0x7fdfe01420d0 con 0x7fdfe0071de0 2024-03-29T11:25:07.097 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.112+0000 7fdfde7fc640 1 -- 172.21.15.17:0/2120079874 >> v1:172.21.15.17:6790/0 conn(0x7fdfe00721c0 legacy=0x7fdfe013f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:07.097 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.112+0000 7fdfde7fc640 1 -- 172.21.15.17:0/2120079874 >> v1:172.21.15.17:6789/0 conn(0x7fdfe0145ae0 legacy=0x7fdfe0145f50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:07.099 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.113+0000 7fdfde7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:25:07.099 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.113+0000 7fdfde7fc640 1 monclient: found mon.noname-c 2024-03-29T11:25:07.099 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.113+0000 7fdfde7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:07.099 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.113+0000 7fdfde7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6789/0 2024-03-29T11:25:07.099 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.113+0000 7fdfde7fc640 1 -- 172.21.15.17:0/2120079874 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfe00b0590 con 0x7fdfe0071de0 2024-03-29T11:25:07.099 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.113+0000 7fdfde7fc640 10 monclient: _finish_auth 0 2024-03-29T11:25:07.099 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.113+0000 7fdfde7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:25:07.099 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.113+0000 7fdfde7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:07.099 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.113+0000 7fdfde7fc640 1 -- 172.21.15.17:0/2120079874 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fdfd0001670 con 0x7fdfe0071de0 2024-03-29T11:25:07.100 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.113+0000 7fdfde7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:07.100 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.113+0000 7fdfde7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:07.100 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.113+0000 7fdfde7fc640 10 monclient: dump: 2024-03-29T11:25:07.100 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:25:07.100 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:25:07.100 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:07.100 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:07.100 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:07.100 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:07.100 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:25:07.100 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:25:07.100 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:25:07.100 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:07.100 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.113+0000 7fdfe6006640 5 monclient: authenticate success, global_id 4468 2024-03-29T11:25:07.101 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.113+0000 7fdfe6006640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:07.101 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.113+0000 7fdfde7fc640 1 -- 172.21.15.17:0/2120079874 <== mon.1 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdfd0001890 con 0x7fdfe0071de0 2024-03-29T11:25:07.101 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.113+0000 7fdfde7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:07.101 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.113+0000 7fdfde7fc640 1 -- 172.21.15.17:0/2120079874 <== mon.1 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fdfd0001ba0 con 0x7fdfe0071de0 2024-03-29T11:25:07.101 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.113+0000 7fdfe6006640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:07.101 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.113+0000 7fdfe6006640 4 set_mon_vals no callback set 2024-03-29T11:25:07.101 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.114+0000 7fdfde7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:07.101 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.114+0000 7fdfde7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:07.101 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.114+0000 7fdfde7fc640 10 monclient: dump: 2024-03-29T11:25:07.101 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:25:07.101 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:25:07.101 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:07.101 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:07.101 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:07.101 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:07.102 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:25:07.102 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:25:07.102 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:25:07.102 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:07.102 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.114+0000 7fdfe6006640 10 monclient: shutdown 2024-03-29T11:25:07.102 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.114+0000 7fdfe6006640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:07.102 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.114+0000 7fdfe6006640 1 -- 172.21.15.17:0/2120079874 >> v1:172.21.15.94:6789/0 conn(0x7fdfe0071de0 legacy=0x7fdfe00aed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:07.102 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.115+0000 7fdfe6006640 1 -- 172.21.15.17:0/2120079874 shutdown_connections 2024-03-29T11:25:07.102 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.115+0000 7fdfe6006640 1 -- 172.21.15.17:0/2120079874 >> 172.21.15.17:0/2120079874 conn(0x7fdfe006b660 msgr2=0x7fdfe006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:07.102 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.115+0000 7fdfe6006640 1 -- 172.21.15.17:0/2120079874 shutdown_connections 2024-03-29T11:25:07.102 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.115+0000 7fdfe6006640 1 -- 172.21.15.17:0/2120079874 wait complete. 2024-03-29T11:25:07.103 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.117+0000 7fdfe6006640 10 monclient: build_initial_monmap 2024-03-29T11:25:07.103 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.117+0000 7fdfe6006640 10 monclient: monmap: 2024-03-29T11:25:07.103 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:25:07.103 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:07.103 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:25:07.103 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:25:07.103 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:07.103 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:07.103 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:25:07.103 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:25:07.104 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:25:07.104 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:07.104 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.117+0000 7fdfe6006640 1 librados: starting msgr at 2024-03-29T11:25:07.105 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.118+0000 7fdfe6006640 1 librados: starting objecter 2024-03-29T11:25:07.105 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.118+0000 7fdfe6006640 1 Processor -- start 2024-03-29T11:25:07.106 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.119+0000 7fdfe6006640 1 -- start start 2024-03-29T11:25:07.106 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.119+0000 7fdfe6006640 1 librados: setting wanted keys 2024-03-29T11:25:07.106 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.119+0000 7fdfe6006640 1 librados: calling monclient init 2024-03-29T11:25:07.106 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.119+0000 7fdfe6006640 10 monclient: init 2024-03-29T11:25:07.106 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.119+0000 7fdfe6006640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:07.106 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.119+0000 7fdfe6006640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:25:07.106 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.120+0000 7fdfe6006640 10 monclient(hunting): picked mon.noname-a con 0x7fdfe0071de0 addr v1:172.21.15.17:6789/0 2024-03-29T11:25:07.106 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.120+0000 7fdfe6006640 10 monclient(hunting): picked mon.noname-c con 0x7fdfe0145ae0 addr v1:172.21.15.17:6790/0 2024-03-29T11:25:07.106 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.120+0000 7fdfe6006640 10 monclient(hunting): picked mon.noname-b con 0x7fdfe0135b60 addr v1:172.21.15.94:6789/0 2024-03-29T11:25:07.106 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.120+0000 7fdfe6006640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfe01325c0 con 0x7fdfe0071de0 2024-03-29T11:25:07.106 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.120+0000 7fdfe6006640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfe0133110 con 0x7fdfe0135b60 2024-03-29T11:25:07.106 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.120+0000 7fdfe6006640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfe0132c30 con 0x7fdfe0145ae0 2024-03-29T11:25:07.106 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.120+0000 7fdfe6006640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:07.106 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.120+0000 7fdfe6006640 10 monclient(hunting): authenticate will time out at 1213.325707s 2024-03-29T11:25:07.107 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.124+0000 7fdfdf7fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fdfe0071de0 0x7fdfe0131eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:55171/0 (socket says 172.21.15.17:55171) 2024-03-29T11:25:07.107 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.124+0000 7fdfdf7fe640 1 -- 172.21.15.17:0/2112561691 learned_addr learned my addr 172.21.15.17:0/2112561691 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:25:07.107 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.124+0000 7fdfc7fff640 10 client.?.objecter ms_handle_connect 0x7fdfe0071de0 2024-03-29T11:25:07.107 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.124+0000 7fdfc7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:07.113 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.126+0000 7fdfc7fff640 10 client.?.objecter ms_handle_connect 0x7fdfe0145ae0 2024-03-29T11:25:07.113 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.126+0000 7fdfc7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:07.113 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.126+0000 7fdfc7fff640 1 -- 172.21.15.17:0/2112561691 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 268844994 0 0) 0x7fdfe0132c30 con 0x7fdfe0145ae0 2024-03-29T11:25:07.113 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.126+0000 7fdfc7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:07.114 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.126+0000 7fdfc7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:07.114 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.126+0000 7fdfc7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:07.114 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.126+0000 7fdfc7fff640 10 monclient(hunting): my global_id is 4487 2024-03-29T11:25:07.114 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.126+0000 7fdfc7fff640 1 -- 172.21.15.17:0/2112561691 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfcc004eb0 con 0x7fdfe0145ae0 2024-03-29T11:25:07.114 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.127+0000 7fdfc7fff640 1 -- 172.21.15.17:0/2112561691 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4236452309 0 0) 0x7fdfe01325c0 con 0x7fdfe0071de0 2024-03-29T11:25:07.114 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.127+0000 7fdfc7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:07.114 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.127+0000 7fdfc7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:07.114 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.127+0000 7fdfc7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:07.114 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.127+0000 7fdfc7fff640 10 monclient(hunting): my global_id is 4437 2024-03-29T11:25:07.114 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.127+0000 7fdfc7fff640 1 -- 172.21.15.17:0/2112561691 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfe0132c30 con 0x7fdfe0071de0 2024-03-29T11:25:07.114 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.127+0000 7fdfc7fff640 10 client.?.objecter ms_handle_connect 0x7fdfe0135b60 2024-03-29T11:25:07.114 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.127+0000 7fdfc7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:07.115 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.127+0000 7fdfc7fff640 1 -- 172.21.15.17:0/2112561691 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2593729577 0 0) 0x7fdfe0133110 con 0x7fdfe0135b60 2024-03-29T11:25:07.115 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.127+0000 7fdfc7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:07.115 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.128+0000 7fdfc7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:07.115 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.128+0000 7fdfc7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:07.115 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.128+0000 7fdfc7fff640 10 monclient(hunting): my global_id is 4471 2024-03-29T11:25:07.115 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.128+0000 7fdfc7fff640 1 -- 172.21.15.17:0/2112561691 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfe01325c0 con 0x7fdfe0135b60 2024-03-29T11:25:07.115 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.128+0000 7fdfc7fff640 1 -- 172.21.15.17:0/2112561691 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1089977735 0 0) 0x7fdfe0132c30 con 0x7fdfe0071de0 2024-03-29T11:25:07.115 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.128+0000 7fdfc7fff640 1 -- 172.21.15.17:0/2112561691 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdfe0133110 con 0x7fdfe0071de0 2024-03-29T11:25:07.115 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.128+0000 7fdfc7fff640 1 -- 172.21.15.17:0/2112561691 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fdfd0001530 con 0x7fdfe0071de0 2024-03-29T11:25:07.115 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.128+0000 7fdfc7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:07.115 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.128+0000 7fdfc7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:07.115 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.128+0000 7fdfc7fff640 10 monclient(hunting): dump: 2024-03-29T11:25:07.115 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:25:07.115 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:25:07.116 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:07.116 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:07.116 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:07.116 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:07.116 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:25:07.116 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:25:07.116 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:25:07.116 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:07.116 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.128+0000 7fdfc7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:07.116 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.128+0000 7fdfc7fff640 1 -- 172.21.15.17:0/2112561691 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1040189387 0 0) 0x7fdfcc004eb0 con 0x7fdfe0145ae0 2024-03-29T11:25:07.116 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.128+0000 7fdfc7fff640 1 -- 172.21.15.17:0/2112561691 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdfe0132c30 con 0x7fdfe0145ae0 2024-03-29T11:25:07.116 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.128+0000 7fdfc7fff640 1 -- 172.21.15.17:0/2112561691 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fdfd4001a70 con 0x7fdfe0145ae0 2024-03-29T11:25:07.116 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.128+0000 7fdfc7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:07.116 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.128+0000 7fdfc7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:07.116 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.128+0000 7fdfc7fff640 10 monclient(hunting): dump: 2024-03-29T11:25:07.117 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:25:07.117 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:25:07.117 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:07.117 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:07.117 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:07.117 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:07.117 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:25:07.117 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:25:07.117 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:25:07.117 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:07.117 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.129+0000 7fdfc7fff640 1 -- 172.21.15.17:0/2112561691 <== mon.0 v1:172.21.15.17:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 392431817 0 0) 0x7fdfe0133110 con 0x7fdfe0071de0 2024-03-29T11:25:07.117 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.129+0000 7fdfc7fff640 1 -- 172.21.15.17:0/2112561691 >> v1:172.21.15.17:6790/0 conn(0x7fdfe0145ae0 legacy=0x7fdfe0130760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:07.117 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.129+0000 7fdfc7fff640 1 -- 172.21.15.17:0/2112561691 >> v1:172.21.15.94:6789/0 conn(0x7fdfe0135b60 legacy=0x7fdfe01da350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:07.117 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.129+0000 7fdfc7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:25:07.117 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.129+0000 7fdfc7fff640 1 monclient: found mon.a 2024-03-29T11:25:07.118 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.129+0000 7fdfc7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:07.118 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.129+0000 7fdfc7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:25:07.118 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.129+0000 7fdfc7fff640 1 -- 172.21.15.17:0/2112561691 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfe0134310 con 0x7fdfe0071de0 2024-03-29T11:25:07.118 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.129+0000 7fdfc7fff640 10 monclient: _finish_auth 0 2024-03-29T11:25:07.118 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.129+0000 7fdfc7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:25:07.118 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.129+0000 7fdfc7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:07.118 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.129+0000 7fdfe6006640 5 monclient: authenticate success, global_id 4437 2024-03-29T11:25:07.118 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.129+0000 7fdfe6006640 10 monclient: _renew_subs 2024-03-29T11:25:07.118 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.129+0000 7fdfe6006640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:25:07.118 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.129+0000 7fdfe6006640 1 -- 172.21.15.17:0/2112561691 --> v1:172.21.15.17:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfe01360a0 con 0x7fdfe0071de0 2024-03-29T11:25:07.119 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.129+0000 7fdfe6006640 10 client.4437.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:07.119 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.129+0000 7fdfe6006640 10 monclient: _renew_subs 2024-03-29T11:25:07.120 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.129+0000 7fdfe6006640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:25:07.120 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.129+0000 7fdfe6006640 1 -- 172.21.15.17:0/2112561691 --> v1:172.21.15.17:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfe01310a0 con 0x7fdfe0071de0 2024-03-29T11:25:07.120 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.129+0000 7fdfe6006640 1 librados: init done 2024-03-29T11:25:07.120 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.130+0000 7fdfc7fff640 1 -- 172.21.15.17:0/2112561691 <== mon.0 v1:172.21.15.17:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdfd00016d0 con 0x7fdfe0071de0 2024-03-29T11:25:07.120 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.130+0000 7fdfc7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:07.120 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.130+0000 7fdfc7fff640 1 -- 172.21.15.17:0/2112561691 <== mon.0 v1:172.21.15.17:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fdfd0004c70 con 0x7fdfe0071de0 2024-03-29T11:25:07.120 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.130+0000 7fdfc7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:07.120 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.130+0000 7fdfc7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:07.120 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.130+0000 7fdfc7fff640 10 monclient: dump: 2024-03-29T11:25:07.120 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:25:07.120 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:25:07.120 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:07.120 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:07.120 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:07.121 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:07.121 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:25:07.121 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:25:07.121 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:25:07.121 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:07.121 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.133+0000 7fdfdd7fa640 4 set_mon_vals no callback set 2024-03-29T11:25:07.121 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.133+0000 7fdfe6006640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:07.121 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.134+0000 7fdfc7fff640 1 -- 172.21.15.17:0/2112561691 <== mon.0 v1:172.21.15.17:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7fdfd0004dd0 con 0x7fdfe0071de0 2024-03-29T11:25:07.121 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.135+0000 7fdfc7fff640 1 -- 172.21.15.17:0/2112561691 <== mon.0 v1:172.21.15.17:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7fdfd0084ab0 con 0x7fdfe0071de0 2024-03-29T11:25:07.121 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.136+0000 7fdfc7fff640 10 client.4437.objecter ms_dispatch 0x7fdfe0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:07.121 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.136+0000 7fdfc7fff640 3 client.4437.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:07.121 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.136+0000 7fdfc7fff640 3 client.4437.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:07.121 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.136+0000 7fdfc7fff640 20 client.4437.objecter dump_active .. 0 homeless 2024-03-29T11:25:07.121 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.136+0000 7fdfe6006640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:07.122 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.136+0000 7fdfe6006640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:25:07.122 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.136+0000 7fdfe6006640 1 -- 172.21.15.17:0/2112561691 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfc0005080 con 0x7fdfe0071de0 2024-03-29T11:25:07.122 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.137+0000 7fdfdeffd640 10 monclient: get_auth_request con 0x7fdfcc06df70 auth_method 0 2024-03-29T11:25:07.123 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.140+0000 7fdfc7fff640 10 client.4437.objecter ms_handle_connect 0x7fdfcc06df70 2024-03-29T11:25:07.129 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.146+0000 7fdfc7fff640 1 -- 172.21.15.17:0/2112561691 <== mon.0 v1:172.21.15.17:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdfd0050390 con 0x7fdfe0071de0 2024-03-29T11:25:07.129 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.146+0000 7fdfc7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:07.129 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.146+0000 7fdfc7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:07.452 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.468+0000 7fdfe6006640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:07.452 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.468+0000 7fdfe6006640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:07.452 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.468+0000 7fdfe6006640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:25:07.452 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.468+0000 7fdfe6006640 1 -- 172.21.15.17:0/2112561691 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdfc0005330 con 0x7fdfe0071de0 2024-03-29T11:25:07.455 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.472+0000 7fdfc7fff640 1 -- 172.21.15.17:0/2112561691 <== mon.0 v1:172.21.15.17:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7fdfd00522c0 con 0x7fdfe0071de0 2024-03-29T11:25:07.455 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.472+0000 7fdfc7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:07.455 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.472+0000 7fdfc7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:07.455 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:25:07.455 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:07.458 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.475+0000 7fdfe6006640 10 librados: watch_flush enter 2024-03-29T11:25:07.458 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.475+0000 7fdfe6006640 10 librados: watch_flush exit 2024-03-29T11:25:07.458 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.475+0000 7fdfe6006640 20 client.4437.objecter shutdown clearing up homeless session... 2024-03-29T11:25:07.458 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.475+0000 7fdfe6006640 10 client.4437.objecter successfully canceled tick 2024-03-29T11:25:07.459 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.476+0000 7fdfe6006640 1 -- 172.21.15.17:0/2112561691 >> v1:172.21.15.17:6816/3236185052 conn(0x7fdfcc06df70 legacy=0x7fdfcc070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:07.459 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.476+0000 7fdfe6006640 10 monclient: shutdown 2024-03-29T11:25:07.459 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.476+0000 7fdfe6006640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:07.459 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.476+0000 7fdfe6006640 1 -- 172.21.15.17:0/2112561691 >> v1:172.21.15.17:6789/0 conn(0x7fdfe0071de0 legacy=0x7fdfe0131eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:07.460 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.477+0000 7fdfe6006640 1 -- 172.21.15.17:0/2112561691 shutdown_connections 2024-03-29T11:25:07.460 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.477+0000 7fdfe6006640 1 -- 172.21.15.17:0/2112561691 >> 172.21.15.17:0/2112561691 conn(0x7fdfe006b660 msgr2=0x7fdfe00b3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:07.461 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.478+0000 7fdfe6006640 1 -- 172.21.15.17:0/2112561691 shutdown_connections 2024-03-29T11:25:07.461 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.478+0000 7fdfe6006640 1 -- 172.21.15.17:0/2112561691 wait complete. 2024-03-29T11:25:07.461 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:07.478+0000 7fdfe6006640 1 librados: shutdown 2024-03-29T11:25:07.480 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:14.287 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:14.439 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.455+0000 7fdd2d16d640 10 monclient: get_monmap_and_config 2024-03-29T11:25:14.439 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.455+0000 7fdd2d16d640 10 monclient: build_initial_monmap 2024-03-29T11:25:14.439 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.455+0000 7fdd2d16d640 10 monclient: monmap: 2024-03-29T11:25:14.439 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:25:14.439 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:25:14.439 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:25:14.456696+0000 2024-03-29T11:25:14.440 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:25:14.456696+0000 2024-03-29T11:25:14.440 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:14.440 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:14.440 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:25:14.440 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:25:14.440 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:25:14.440 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:14.440 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.457+0000 7fdd2d16d640 1 Processor -- start 2024-03-29T11:25:14.441 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.457+0000 7fdd2d16d640 1 -- start start 2024-03-29T11:25:14.441 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.457+0000 7fdd2d16d640 10 monclient: init 2024-03-29T11:25:14.444 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.458+0000 7fdd2d16d640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:14.445 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.458+0000 7fdd2d16d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:25:14.445 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.458+0000 7fdd2d16d640 10 monclient(hunting): picked mon.noname-a con 0x7fdd20006e30 addr v1:172.21.15.17:6789/0 2024-03-29T11:25:14.445 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.458+0000 7fdd2d16d640 10 monclient(hunting): picked mon.noname-c con 0x7fdd20035e00 addr v1:172.21.15.94:6789/0 2024-03-29T11:25:14.445 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.458+0000 7fdd2d16d640 10 monclient(hunting): picked mon.noname-b con 0x7fdd200c9670 addr v1:172.21.15.17:6790/0 2024-03-29T11:25:14.445 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.458+0000 7fdd2d16d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd20035970 con 0x7fdd20006e30 2024-03-29T11:25:14.445 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.458+0000 7fdd2d16d640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd20036280 con 0x7fdd20035e00 2024-03-29T11:25:14.445 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.458+0000 7fdd2d16d640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd20035640 con 0x7fdd200c9670 2024-03-29T11:25:14.445 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.458+0000 7fdd2d16d640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:14.445 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.458+0000 7fdd2d16d640 10 monclient(hunting): authenticate will time out at 1220.663482s 2024-03-29T11:25:14.446 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.462+0000 7fdd27fff640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fdd20006e30 0x7fdd200cbcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:50264/0 (socket says 172.21.15.17:50264) 2024-03-29T11:25:14.446 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.462+0000 7fdd277fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fdd20035e00 0x7fdd200c8f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:55781/0 (socket says 172.21.15.17:55781) 2024-03-29T11:25:14.446 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.462+0000 7fdd27fff640 1 -- 172.21.15.17:0/3966122532 learned_addr learned my addr 172.21.15.17:0/3966122532 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:25:14.446 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.463+0000 7fdd26ffd640 1 -- 172.21.15.17:0/3966122532 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1516452879 0 0) 0x7fdd20036280 con 0x7fdd20035e00 2024-03-29T11:25:14.447 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.463+0000 7fdd26ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:14.447 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.463+0000 7fdd26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:14.447 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.463+0000 7fdd26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:14.447 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.463+0000 7fdd26ffd640 10 monclient(hunting): my global_id is 4474 2024-03-29T11:25:14.447 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.463+0000 7fdd26ffd640 1 -- 172.21.15.17:0/3966122532 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd18003520 con 0x7fdd20035e00 2024-03-29T11:25:14.447 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.463+0000 7fdd26ffd640 1 -- 172.21.15.17:0/3966122532 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3597200528 0 0) 0x7fdd20035970 con 0x7fdd20006e30 2024-03-29T11:25:14.447 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.463+0000 7fdd26ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:14.447 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.463+0000 7fdd26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:14.447 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.463+0000 7fdd26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:14.447 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.463+0000 7fdd26ffd640 10 monclient(hunting): my global_id is 4440 2024-03-29T11:25:14.447 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.463+0000 7fdd26ffd640 1 -- 172.21.15.17:0/3966122532 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd20036280 con 0x7fdd20006e30 2024-03-29T11:25:14.448 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.464+0000 7fdd26ffd640 1 -- 172.21.15.17:0/3966122532 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 807733484 0 0) 0x7fdd18003520 con 0x7fdd20035e00 2024-03-29T11:25:14.448 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.464+0000 7fdd26ffd640 1 -- 172.21.15.17:0/3966122532 >> v1:172.21.15.17:6790/0 conn(0x7fdd200c9670 legacy=0x7fdd200d7560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:14.448 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.465+0000 7fdd26ffd640 1 -- 172.21.15.17:0/3966122532 >> v1:172.21.15.17:6789/0 conn(0x7fdd20006e30 legacy=0x7fdd200cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:14.449 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.465+0000 7fdd26ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:25:14.449 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.465+0000 7fdd26ffd640 1 monclient: found mon.noname-c 2024-03-29T11:25:14.449 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.465+0000 7fdd26ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:14.449 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.465+0000 7fdd26ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6789/0 2024-03-29T11:25:14.449 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.465+0000 7fdd26ffd640 1 -- 172.21.15.17:0/3966122532 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd20034e20 con 0x7fdd20035e00 2024-03-29T11:25:14.449 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.465+0000 7fdd26ffd640 10 monclient: _finish_auth 0 2024-03-29T11:25:14.449 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.465+0000 7fdd26ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:25:14.449 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.465+0000 7fdd26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:14.449 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.465+0000 7fdd26ffd640 1 -- 172.21.15.17:0/3966122532 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fdd14001690 con 0x7fdd20035e00 2024-03-29T11:25:14.449 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.465+0000 7fdd26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:14.449 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.465+0000 7fdd26ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:14.449 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.465+0000 7fdd26ffd640 10 monclient: dump: 2024-03-29T11:25:14.449 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:25:14.449 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:25:14.450 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:14.450 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:14.450 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:14.450 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:14.450 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:25:14.450 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:25:14.450 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:25:14.450 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:14.450 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.465+0000 7fdd2d16d640 5 monclient: authenticate success, global_id 4474 2024-03-29T11:25:14.450 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.465+0000 7fdd2d16d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:14.451 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.465+0000 7fdd26ffd640 1 -- 172.21.15.17:0/3966122532 <== mon.1 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd14001860 con 0x7fdd20035e00 2024-03-29T11:25:14.451 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.465+0000 7fdd26ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:14.451 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.465+0000 7fdd2d16d640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:14.451 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.465+0000 7fdd2d16d640 4 set_mon_vals no callback set 2024-03-29T11:25:14.451 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.466+0000 7fdd26ffd640 1 -- 172.21.15.17:0/3966122532 <== mon.1 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fdd14001c70 con 0x7fdd20035e00 2024-03-29T11:25:14.451 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.466+0000 7fdd26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:14.451 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.466+0000 7fdd26ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:14.452 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.466+0000 7fdd26ffd640 10 monclient: dump: 2024-03-29T11:25:14.452 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:25:14.452 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:25:14.452 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:14.452 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:14.452 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:14.452 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:14.452 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:25:14.452 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:25:14.452 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:25:14.452 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:14.452 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.466+0000 7fdd2d16d640 10 monclient: shutdown 2024-03-29T11:25:14.452 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.466+0000 7fdd2d16d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:14.452 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.466+0000 7fdd2d16d640 1 -- 172.21.15.17:0/3966122532 >> v1:172.21.15.94:6789/0 conn(0x7fdd20035e00 legacy=0x7fdd200c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:14.452 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.467+0000 7fdd2d16d640 1 -- 172.21.15.17:0/3966122532 shutdown_connections 2024-03-29T11:25:14.453 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.467+0000 7fdd2d16d640 1 -- 172.21.15.17:0/3966122532 >> 172.21.15.17:0/3966122532 conn(0x7fdd20005c00 msgr2=0x7fdd20006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:14.453 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.467+0000 7fdd2d16d640 1 -- 172.21.15.17:0/3966122532 shutdown_connections 2024-03-29T11:25:14.453 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.467+0000 7fdd2d16d640 1 -- 172.21.15.17:0/3966122532 wait complete. 2024-03-29T11:25:14.453 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.468+0000 7fdd2d16d640 10 monclient: build_initial_monmap 2024-03-29T11:25:14.453 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.468+0000 7fdd2d16d640 10 monclient: monmap: 2024-03-29T11:25:14.453 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:25:14.453 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:14.453 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:25:14.453 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:25:14.453 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:14.453 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:14.453 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:25:14.453 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:25:14.453 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:25:14.453 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:14.454 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.468+0000 7fdd2d16d640 1 librados: starting msgr at 2024-03-29T11:25:14.454 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.468+0000 7fdd2d16d640 1 librados: starting objecter 2024-03-29T11:25:14.454 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.468+0000 7fdd2d16d640 1 Processor -- start 2024-03-29T11:25:14.458 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.469+0000 7fdd2d16d640 1 -- start start 2024-03-29T11:25:14.458 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.469+0000 7fdd2d16d640 1 librados: setting wanted keys 2024-03-29T11:25:14.458 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.469+0000 7fdd2d16d640 1 librados: calling monclient init 2024-03-29T11:25:14.458 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.469+0000 7fdd2d16d640 10 monclient: init 2024-03-29T11:25:14.458 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.469+0000 7fdd2d16d640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:14.459 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.469+0000 7fdd2d16d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:25:14.459 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.469+0000 7fdd2d16d640 10 monclient(hunting): picked mon.noname-b con 0x7fdd20006e30 addr v1:172.21.15.94:6789/0 2024-03-29T11:25:14.459 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.469+0000 7fdd2d16d640 10 monclient(hunting): picked mon.noname-c con 0x7fdd20035e00 addr v1:172.21.15.17:6790/0 2024-03-29T11:25:14.459 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.469+0000 7fdd2d16d640 10 monclient(hunting): picked mon.noname-a con 0x7fdd200c9670 addr v1:172.21.15.17:6789/0 2024-03-29T11:25:14.459 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.469+0000 7fdd2d16d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd2002d7a0 con 0x7fdd200c9670 2024-03-29T11:25:14.459 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.469+0000 7fdd2d16d640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd20161fe0 con 0x7fdd20006e30 2024-03-29T11:25:14.459 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.469+0000 7fdd2d16d640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd20161be0 con 0x7fdd20035e00 2024-03-29T11:25:14.459 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.469+0000 7fdd2d16d640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:14.459 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.469+0000 7fdd2d16d640 10 monclient(hunting): authenticate will time out at 1220.674704s 2024-03-29T11:25:14.459 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.472+0000 7fdd27fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fdd20006e30 0x7fdd201614d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:55855/0 (socket says 172.21.15.17:55855) 2024-03-29T11:25:14.459 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.472+0000 7fdd27fff640 1 -- 172.21.15.17:0/1266107218 learned_addr learned my addr 172.21.15.17:0/1266107218 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:25:14.459 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.472+0000 7fdd24ff9640 10 client.?.objecter ms_handle_connect 0x7fdd20006e30 2024-03-29T11:25:14.459 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.472+0000 7fdd24ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:14.459 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.472+0000 7fdd24ff9640 1 -- 172.21.15.17:0/1266107218 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 712707263 0 0) 0x7fdd20161fe0 con 0x7fdd20006e30 2024-03-29T11:25:14.460 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.472+0000 7fdd24ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:25:14.460 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.472+0000 7fdd24ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:14.460 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.472+0000 7fdd24ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:14.460 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.472+0000 7fdd24ff9640 10 monclient(hunting): my global_id is 4477 2024-03-29T11:25:14.460 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.472+0000 7fdd24ff9640 1 -- 172.21.15.17:0/1266107218 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd0c004eb0 con 0x7fdd20006e30 2024-03-29T11:25:14.460 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.473+0000 7fdd24ff9640 1 -- 172.21.15.17:0/1266107218 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3211734666 0 0) 0x7fdd0c004eb0 con 0x7fdd20006e30 2024-03-29T11:25:14.460 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.473+0000 7fdd24ff9640 1 -- 172.21.15.17:0/1266107218 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd20161fe0 con 0x7fdd20006e30 2024-03-29T11:25:14.460 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.473+0000 7fdd24ff9640 1 -- 172.21.15.17:0/1266107218 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fdd2804fdd0 con 0x7fdd20006e30 2024-03-29T11:25:14.460 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.473+0000 7fdd24ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:14.460 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.473+0000 7fdd24ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:14.460 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.473+0000 7fdd24ff9640 10 monclient(hunting): dump: 2024-03-29T11:25:14.460 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:25:14.460 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:25:14.460 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:14.460 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:14.461 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:14.461 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:14.461 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:25:14.461 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:25:14.461 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:25:14.461 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:14.461 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.473+0000 7fdd24ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:14.461 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.474+0000 7fdd24ff9640 1 -- 172.21.15.17:0/1266107218 <== mon.1 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1724453595 0 0) 0x7fdd20161fe0 con 0x7fdd20006e30 2024-03-29T11:25:14.461 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.474+0000 7fdd24ff9640 1 -- 172.21.15.17:0/1266107218 >> v1:172.21.15.17:6790/0 conn(0x7fdd20035e00 legacy=0x7fdd200c2d00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:25:14.462 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.476+0000 7fdd24ff9640 1 -- 172.21.15.17:0/1266107218 >> v1:172.21.15.17:6789/0 conn(0x7fdd200c9670 legacy=0x7fdd200ccfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:14.462 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.476+0000 7fdd24ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:25:14.462 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.476+0000 7fdd24ff9640 1 monclient: found mon.b 2024-03-29T11:25:14.462 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.476+0000 7fdd24ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:14.463 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.476+0000 7fdd24ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:25:14.463 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.476+0000 7fdd24ff9640 1 -- 172.21.15.17:0/1266107218 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd20161d70 con 0x7fdd20006e30 2024-03-29T11:25:14.463 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.477+0000 7fdd24ff9640 10 monclient: _finish_auth 0 2024-03-29T11:25:14.463 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.477+0000 7fdd24ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:25:14.463 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.477+0000 7fdd24ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:14.465 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.477+0000 7fdd2d16d640 5 monclient: authenticate success, global_id 4477 2024-03-29T11:25:14.465 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.477+0000 7fdd2d16d640 10 monclient: _renew_subs 2024-03-29T11:25:14.465 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.477+0000 7fdd2d16d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:25:14.465 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.477+0000 7fdd2d16d640 1 -- 172.21.15.17:0/1266107218 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd200c3410 con 0x7fdd20006e30 2024-03-29T11:25:14.465 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.477+0000 7fdd2d16d640 10 client.4477.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:14.465 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.477+0000 7fdd2d16d640 10 monclient: _renew_subs 2024-03-29T11:25:14.465 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.477+0000 7fdd2d16d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:25:14.465 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.477+0000 7fdd2d16d640 1 -- 172.21.15.17:0/1266107218 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd20032a30 con 0x7fdd20006e30 2024-03-29T11:25:14.466 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.477+0000 7fdd2d16d640 1 librados: init done 2024-03-29T11:25:14.466 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.480+0000 7fdd24ff9640 1 -- 172.21.15.17:0/1266107218 <== mon.1 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd2804f400 con 0x7fdd20006e30 2024-03-29T11:25:14.466 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.480+0000 7fdd24ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:14.466 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.480+0000 7fdd25ffb640 4 set_mon_vals no callback set 2024-03-29T11:25:14.466 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.480+0000 7fdd127fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:14.466 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.481+0000 7fdd24ff9640 1 -- 172.21.15.17:0/1266107218 <== mon.1 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fdd28075f10 con 0x7fdd20006e30 2024-03-29T11:25:14.466 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.481+0000 7fdd24ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:14.466 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.481+0000 7fdd24ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:14.466 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.481+0000 7fdd24ff9640 10 monclient: dump: 2024-03-29T11:25:14.466 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:25:14.466 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:25:14.466 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:14.466 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:14.467 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:14.467 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:14.467 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:25:14.467 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:25:14.467 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:25:14.467 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:14.467 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.481+0000 7fdd24ff9640 1 -- 172.21.15.17:0/1266107218 <== mon.1 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7fdd28067bf0 con 0x7fdd20006e30 2024-03-29T11:25:14.467 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.481+0000 7fdd127fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:14.467 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.481+0000 7fdd127fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:25:14.467 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.481+0000 7fdd127fc640 1 -- 172.21.15.17:0/1266107218 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd08005080 con 0x7fdd20006e30 2024-03-29T11:25:14.468 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.482+0000 7fdd24ff9640 1 -- 172.21.15.17:0/1266107218 <== mon.1 v1:172.21.15.94:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7fdd280ff7b0 con 0x7fdd20006e30 2024-03-29T11:25:14.468 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.482+0000 7fdd24ff9640 10 client.4477.objecter ms_dispatch 0x7fdd28000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:14.468 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.482+0000 7fdd24ff9640 3 client.4477.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:14.468 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.482+0000 7fdd24ff9640 3 client.4477.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:14.468 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.482+0000 7fdd24ff9640 20 client.4477.objecter dump_active .. 0 homeless 2024-03-29T11:25:14.468 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.483+0000 7fdd277fe640 10 monclient: get_auth_request con 0x7fdd0c06dc60 auth_method 0 2024-03-29T11:25:14.469 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.484+0000 7fdd24ff9640 10 client.4477.objecter ms_handle_connect 0x7fdd0c06dc60 2024-03-29T11:25:14.474 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.491+0000 7fdd24ff9640 1 -- 172.21.15.17:0/1266107218 <== mon.1 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdd280cb170 con 0x7fdd20006e30 2024-03-29T11:25:14.474 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.491+0000 7fdd24ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:14.475 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.491+0000 7fdd24ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:14.806 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.822+0000 7fdd127fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:14.806 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.822+0000 7fdd127fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:14.806 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.822+0000 7fdd127fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:25:14.806 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.822+0000 7fdd127fc640 1 -- 172.21.15.17:0/1266107218 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd08005330 con 0x7fdd20006e30 2024-03-29T11:25:14.808 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.824+0000 7fdd24ff9640 1 -- 172.21.15.17:0/1266107218 <== mon.1 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7fdd280cd0a0 con 0x7fdd20006e30 2024-03-29T11:25:14.808 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.824+0000 7fdd24ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:14.808 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.824+0000 7fdd24ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:14.808 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:25:14.808 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:14.811 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.828+0000 7fdd127fc640 10 librados: watch_flush enter 2024-03-29T11:25:14.813 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.829+0000 7fdd127fc640 10 librados: watch_flush exit 2024-03-29T11:25:14.813 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.829+0000 7fdd127fc640 20 client.4477.objecter shutdown clearing up homeless session... 2024-03-29T11:25:14.813 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.829+0000 7fdd127fc640 10 client.4477.objecter successfully canceled tick 2024-03-29T11:25:14.813 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.829+0000 7fdd127fc640 1 -- 172.21.15.17:0/1266107218 >> v1:172.21.15.17:6816/3236185052 conn(0x7fdd0c06dc60 legacy=0x7fdd0c070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:14.814 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.829+0000 7fdd127fc640 10 monclient: shutdown 2024-03-29T11:25:14.814 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.830+0000 7fdd127fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:14.814 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.830+0000 7fdd127fc640 1 -- 172.21.15.17:0/1266107218 >> v1:172.21.15.94:6789/0 conn(0x7fdd20006e30 legacy=0x7fdd201614d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:14.814 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.830+0000 7fdd127fc640 1 -- 172.21.15.17:0/1266107218 shutdown_connections 2024-03-29T11:25:14.814 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.830+0000 7fdd127fc640 1 -- 172.21.15.17:0/1266107218 >> 172.21.15.17:0/1266107218 conn(0x7fdd20005c00 msgr2=0x7fdd200c8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:14.817 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.833+0000 7fdd127fc640 1 -- 172.21.15.17:0/1266107218 shutdown_connections 2024-03-29T11:25:14.817 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.833+0000 7fdd127fc640 1 -- 172.21.15.17:0/1266107218 wait complete. 2024-03-29T11:25:14.817 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:14.834+0000 7fdd127fc640 1 librados: shutdown 2024-03-29T11:25:14.837 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:21.644 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:21.802 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.815+0000 7f45b3da3640 10 monclient: get_monmap_and_config 2024-03-29T11:25:21.802 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.815+0000 7f45b3da3640 10 monclient: build_initial_monmap 2024-03-29T11:25:21.802 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.816+0000 7f45b3da3640 10 monclient: monmap: 2024-03-29T11:25:21.802 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:25:21.802 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:25:21.802 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:25:21.816988+0000 2024-03-29T11:25:21.802 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:25:21.816988+0000 2024-03-29T11:25:21.802 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:21.802 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:21.803 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:25:21.803 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:25:21.803 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:25:21.803 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:21.803 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.817+0000 7f45b3da3640 1 Processor -- start 2024-03-29T11:25:21.803 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.818+0000 7f45b3da3640 1 -- start start 2024-03-29T11:25:21.803 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.818+0000 7f45b3da3640 10 monclient: init 2024-03-29T11:25:21.805 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.819+0000 7f45b3da3640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:21.805 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.819+0000 7f45b3da3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:25:21.805 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.819+0000 7f45b3da3640 10 monclient(hunting): picked mon.noname-a con 0x7f45ac0a9750 addr v1:172.21.15.17:6789/0 2024-03-29T11:25:21.805 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.819+0000 7f45b3da3640 10 monclient(hunting): picked mon.noname-c con 0x7f45ac149ac0 addr v1:172.21.15.94:6789/0 2024-03-29T11:25:21.805 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.819+0000 7f45b3da3640 10 monclient(hunting): picked mon.noname-b con 0x7f45ac14fe50 addr v1:172.21.15.17:6790/0 2024-03-29T11:25:21.805 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.819+0000 7f45b3da3640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45ac0aa4a0 con 0x7f45ac0a9750 2024-03-29T11:25:21.805 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.819+0000 7f45b3da3640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45ac14a680 con 0x7f45ac149ac0 2024-03-29T11:25:21.806 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.819+0000 7f45b3da3640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45ac0af290 con 0x7f45ac14fe50 2024-03-29T11:25:21.806 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.819+0000 7f45b3da3640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:21.806 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.819+0000 7f45b3da3640 10 monclient(hunting): authenticate will time out at 1228.024681s 2024-03-29T11:25:21.806 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.819+0000 7f45b133e640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f45ac149ac0 0x7f45ac149f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:44558/0 (socket says 172.21.15.17:44558) 2024-03-29T11:25:21.806 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.819+0000 7f45b133e640 1 -- 172.21.15.17:0/41375515 learned_addr learned my addr 172.21.15.17:0/41375515 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:25:21.806 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.820+0000 7f45b0b3d640 1 -- 172.21.15.17:0/41375515 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2945697130 0 0) 0x7f45ac14a680 con 0x7f45ac149ac0 2024-03-29T11:25:21.806 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.820+0000 7f45b0b3d640 10 monclient(hunting): _negotiate 2024-03-29T11:25:21.806 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.820+0000 7f45b0b3d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:21.806 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.820+0000 7f45b0b3d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:21.806 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.820+0000 7f45b0b3d640 10 monclient(hunting): my global_id is 4480 2024-03-29T11:25:21.806 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.821+0000 7f45b0b3d640 1 -- 172.21.15.17:0/41375515 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4594003520 con 0x7f45ac149ac0 2024-03-29T11:25:21.806 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.821+0000 7f45b0b3d640 1 -- 172.21.15.17:0/41375515 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1903505087 0 0) 0x7f4594003520 con 0x7f45ac149ac0 2024-03-29T11:25:21.807 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.821+0000 7f45b0b3d640 1 -- 172.21.15.17:0/41375515 >> v1:172.21.15.17:6790/0 conn(0x7f45ac14fe50 legacy=0x7f45ac152240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:21.807 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.821+0000 7f45b0b3d640 1 -- 172.21.15.17:0/41375515 >> v1:172.21.15.17:6789/0 conn(0x7f45ac0a9750 legacy=0x7f45ac1436f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:21.807 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.821+0000 7f45b0b3d640 10 monclient: _finish_hunting 0 2024-03-29T11:25:21.807 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.821+0000 7f45b0b3d640 1 monclient: found mon.noname-c 2024-03-29T11:25:21.807 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.821+0000 7f45b0b3d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:21.807 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.821+0000 7f45b0b3d640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6789/0 2024-03-29T11:25:21.807 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.821+0000 7f45b0b3d640 1 -- 172.21.15.17:0/41375515 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45ac0afb40 con 0x7f45ac149ac0 2024-03-29T11:25:21.807 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.821+0000 7f45b0b3d640 10 monclient: _finish_auth 0 2024-03-29T11:25:21.807 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.821+0000 7f45b0b3d640 10 monclient: _check_auth_tickets 2024-03-29T11:25:21.807 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.821+0000 7f45b0b3d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:21.807 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.821+0000 7f45b0b3d640 1 -- 172.21.15.17:0/41375515 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f45a0001690 con 0x7f45ac149ac0 2024-03-29T11:25:21.807 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.821+0000 7f45b0b3d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:21.807 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.821+0000 7f45b0b3d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:21.807 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.821+0000 7f45b0b3d640 10 monclient: dump: 2024-03-29T11:25:21.807 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:25:21.808 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:25:21.808 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:21.808 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:21.808 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:21.808 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:21.808 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:25:21.808 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:25:21.808 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:25:21.808 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:21.808 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.821+0000 7f45b3da3640 5 monclient: authenticate success, global_id 4480 2024-03-29T11:25:21.808 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.821+0000 7f45b3da3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:21.809 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.822+0000 7f45b0b3d640 1 -- 172.21.15.17:0/41375515 <== mon.1 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45a0001860 con 0x7f45ac149ac0 2024-03-29T11:25:21.809 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.822+0000 7f45b0b3d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:21.809 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.822+0000 7f45b3da3640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:21.809 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.822+0000 7f45b3da3640 4 set_mon_vals no callback set 2024-03-29T11:25:21.809 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.822+0000 7f45b0b3d640 1 -- 172.21.15.17:0/41375515 <== mon.1 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f45a0001cb0 con 0x7f45ac149ac0 2024-03-29T11:25:21.809 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.822+0000 7f45b0b3d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:21.809 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.822+0000 7f45b0b3d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:21.810 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.822+0000 7f45b0b3d640 10 monclient: dump: 2024-03-29T11:25:21.810 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:25:21.810 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:25:21.810 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:21.810 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:21.810 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:21.810 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:21.810 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:25:21.810 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:25:21.810 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:25:21.810 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:21.810 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.823+0000 7f45b3da3640 10 monclient: shutdown 2024-03-29T11:25:21.810 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.823+0000 7f45b3da3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:21.810 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.823+0000 7f45b3da3640 1 -- 172.21.15.17:0/41375515 >> v1:172.21.15.94:6789/0 conn(0x7f45ac149ac0 legacy=0x7f45ac149f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:21.811 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.823+0000 7f45b3da3640 1 -- 172.21.15.17:0/41375515 shutdown_connections 2024-03-29T11:25:21.811 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.823+0000 7f45b3da3640 1 -- 172.21.15.17:0/41375515 >> 172.21.15.17:0/41375515 conn(0x7f45ac0b35e0 msgr2=0x7f45ac0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:21.811 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.823+0000 7f45b3da3640 1 -- 172.21.15.17:0/41375515 shutdown_connections 2024-03-29T11:25:21.811 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.823+0000 7f45b3da3640 1 -- 172.21.15.17:0/41375515 wait complete. 2024-03-29T11:25:21.811 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.824+0000 7f45b3da3640 10 monclient: build_initial_monmap 2024-03-29T11:25:21.811 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.824+0000 7f45b3da3640 10 monclient: monmap: 2024-03-29T11:25:21.811 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:25:21.811 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:21.811 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:25:21.811 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:25:21.811 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:21.811 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:21.811 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:25:21.811 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:25:21.811 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:25:21.811 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:21.812 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.824+0000 7f45b3da3640 1 librados: starting msgr at 2024-03-29T11:25:21.812 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.824+0000 7f45b3da3640 1 librados: starting objecter 2024-03-29T11:25:21.812 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.825+0000 7f45b3da3640 1 Processor -- start 2024-03-29T11:25:21.815 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.825+0000 7f45b3da3640 1 -- start start 2024-03-29T11:25:21.816 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.825+0000 7f45b3da3640 1 librados: setting wanted keys 2024-03-29T11:25:21.816 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.825+0000 7f45b3da3640 1 librados: calling monclient init 2024-03-29T11:25:21.816 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.825+0000 7f45b3da3640 10 monclient: init 2024-03-29T11:25:21.816 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.826+0000 7f45b3da3640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:21.816 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.826+0000 7f45b3da3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:25:21.816 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.826+0000 7f45b3da3640 10 monclient(hunting): picked mon.noname-b con 0x7f45ac0a9750 addr v1:172.21.15.94:6789/0 2024-03-29T11:25:21.816 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.826+0000 7f45b3da3640 10 monclient(hunting): picked mon.noname-a con 0x7f45ac149ac0 addr v1:172.21.15.17:6789/0 2024-03-29T11:25:21.816 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.826+0000 7f45b3da3640 10 monclient(hunting): picked mon.noname-c con 0x7f45ac14fe50 addr v1:172.21.15.17:6790/0 2024-03-29T11:25:21.816 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.826+0000 7f45b3da3640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45ac148970 con 0x7f45ac149ac0 2024-03-29T11:25:21.816 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.826+0000 7f45b3da3640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45ac0aca30 con 0x7f45ac0a9750 2024-03-29T11:25:21.816 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.826+0000 7f45b3da3640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45ac13a980 con 0x7f45ac14fe50 2024-03-29T11:25:21.816 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.826+0000 7f45b3da3640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:21.817 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.826+0000 7f45b3da3640 10 monclient(hunting): authenticate will time out at 1228.031156s 2024-03-29T11:25:21.817 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.826+0000 7f45b1b3f640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f45ac0a9750 0x7f45ac13a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:45526/0 (socket says 172.21.15.17:45526) 2024-03-29T11:25:21.817 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.826+0000 7f45b1b3f640 1 -- 172.21.15.17:0/1268483716 learned_addr learned my addr 172.21.15.17:0/1268483716 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:25:21.817 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.828+0000 7f45aa7fc640 10 client.?.objecter ms_handle_connect 0x7f45ac0a9750 2024-03-29T11:25:21.817 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.828+0000 7f45aa7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:21.817 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.828+0000 7f45aa7fc640 1 -- 172.21.15.17:0/1268483716 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3298900161 0 0) 0x7f45ac0aca30 con 0x7f45ac0a9750 2024-03-29T11:25:21.817 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.828+0000 7f45aa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:21.817 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.828+0000 7f45aa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:21.817 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.828+0000 7f45aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:21.817 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.828+0000 7f45aa7fc640 10 monclient(hunting): my global_id is 4483 2024-03-29T11:25:21.817 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.828+0000 7f45aa7fc640 1 -- 172.21.15.17:0/1268483716 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4598004eb0 con 0x7f45ac0a9750 2024-03-29T11:25:21.818 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.828+0000 7f45aa7fc640 10 client.?.objecter ms_handle_connect 0x7f45ac149ac0 2024-03-29T11:25:21.818 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.828+0000 7f45aa7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:21.818 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.828+0000 7f45aa7fc640 1 -- 172.21.15.17:0/1268483716 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2335119106 0 0) 0x7f45ac148970 con 0x7f45ac149ac0 2024-03-29T11:25:21.818 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.828+0000 7f45aa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:21.818 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.828+0000 7f45aa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:21.818 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.828+0000 7f45aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:21.818 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.828+0000 7f45aa7fc640 10 monclient(hunting): my global_id is 4443 2024-03-29T11:25:21.818 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.828+0000 7f45aa7fc640 1 -- 172.21.15.17:0/1268483716 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45ac0aca30 con 0x7f45ac149ac0 2024-03-29T11:25:21.818 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.829+0000 7f45aa7fc640 1 -- 172.21.15.17:0/1268483716 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2509619069 0 0) 0x7f45ac0aca30 con 0x7f45ac149ac0 2024-03-29T11:25:21.818 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.829+0000 7f45aa7fc640 1 -- 172.21.15.17:0/1268483716 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45ac148970 con 0x7f45ac149ac0 2024-03-29T11:25:21.818 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.829+0000 7f45aa7fc640 1 -- 172.21.15.17:0/1268483716 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f45a0001b70 con 0x7f45ac149ac0 2024-03-29T11:25:21.818 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.829+0000 7f45aa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:21.818 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.829+0000 7f45aa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:21.818 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.829+0000 7f45aa7fc640 10 monclient(hunting): dump: 2024-03-29T11:25:21.819 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:25:21.819 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:25:21.819 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:21.819 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:21.819 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:21.819 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:21.819 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:25:21.819 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:25:21.819 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:25:21.819 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:21.819 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.829+0000 7f45aa7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:21.819 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.829+0000 7f45aa7fc640 1 -- 172.21.15.17:0/1268483716 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1747692728 0 0) 0x7f4598004eb0 con 0x7f45ac0a9750 2024-03-29T11:25:21.819 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.829+0000 7f45aa7fc640 1 -- 172.21.15.17:0/1268483716 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45ac0aca30 con 0x7f45ac0a9750 2024-03-29T11:25:21.819 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.829+0000 7f45aa7fc640 10 client.?.objecter ms_handle_connect 0x7f45ac14fe50 2024-03-29T11:25:21.819 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.829+0000 7f45aa7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:21.820 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.829+0000 7f45aa7fc640 1 -- 172.21.15.17:0/1268483716 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f459c001750 con 0x7f45ac0a9750 2024-03-29T11:25:21.820 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.829+0000 7f45aa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:21.820 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.829+0000 7f45aa7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:21.820 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.829+0000 7f45aa7fc640 10 monclient(hunting): dump: 2024-03-29T11:25:21.820 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:25:21.820 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:25:21.820 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:21.820 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:21.820 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:21.820 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:21.820 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:25:21.821 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:25:21.821 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:25:21.821 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:21.821 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.829+0000 7f45aa7fc640 1 -- 172.21.15.17:0/1268483716 <== mon.1 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3640411320 0 0) 0x7f45ac0aca30 con 0x7f45ac0a9750 2024-03-29T11:25:21.821 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.829+0000 7f45aa7fc640 1 -- 172.21.15.17:0/1268483716 >> v1:172.21.15.17:6790/0 conn(0x7f45ac14fe50 legacy=0x7f45ac148e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:21.821 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.829+0000 7f45aa7fc640 1 -- 172.21.15.17:0/1268483716 >> v1:172.21.15.17:6789/0 conn(0x7f45ac149ac0 legacy=0x7f45ac1db940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:21.821 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.829+0000 7f45aa7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:25:21.821 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.829+0000 7f45aa7fc640 1 monclient: found mon.b 2024-03-29T11:25:21.821 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.829+0000 7f45aa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:21.821 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.829+0000 7f45aa7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:25:21.821 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.829+0000 7f45aa7fc640 1 -- 172.21.15.17:0/1268483716 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45ac1dc050 con 0x7f45ac0a9750 2024-03-29T11:25:21.821 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.829+0000 7f45aa7fc640 10 monclient: _finish_auth 0 2024-03-29T11:25:21.821 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.829+0000 7f45aa7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:25:21.821 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.829+0000 7f45aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:21.821 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.829+0000 7f45b3da3640 5 monclient: authenticate success, global_id 4483 2024-03-29T11:25:21.822 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.829+0000 7f45b3da3640 10 monclient: _renew_subs 2024-03-29T11:25:21.822 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.829+0000 7f45b3da3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:25:21.822 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.829+0000 7f45b3da3640 1 -- 172.21.15.17:0/1268483716 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45ac1dc240 con 0x7f45ac0a9750 2024-03-29T11:25:21.822 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.829+0000 7f45b3da3640 10 client.4483.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:21.822 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.829+0000 7f45b3da3640 10 monclient: _renew_subs 2024-03-29T11:25:21.822 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.829+0000 7f45b3da3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:25:21.822 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.829+0000 7f45b3da3640 1 -- 172.21.15.17:0/1268483716 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f45ac0ad130 con 0x7f45ac0a9750 2024-03-29T11:25:21.822 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.829+0000 7f45b3da3640 1 librados: init done 2024-03-29T11:25:21.822 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.830+0000 7f45aa7fc640 1 -- 172.21.15.17:0/1268483716 <== mon.1 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f459c0018b0 con 0x7f45ac0a9750 2024-03-29T11:25:21.822 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.830+0000 7f45aa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:21.822 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.830+0000 7f45abfff640 4 set_mon_vals no callback set 2024-03-29T11:25:21.822 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.831+0000 7f45b3da3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:21.822 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.831+0000 7f45aa7fc640 1 -- 172.21.15.17:0/1268483716 <== mon.1 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f459c004a00 con 0x7f45ac0a9750 2024-03-29T11:25:21.822 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.831+0000 7f45b3da3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:21.823 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.831+0000 7f45b3da3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:25:21.823 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.831+0000 7f45b3da3640 1 -- 172.21.15.17:0/1268483716 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f458c005080 con 0x7f45ac0a9750 2024-03-29T11:25:21.823 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.832+0000 7f45aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:21.824 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.832+0000 7f45aa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:21.824 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.832+0000 7f45aa7fc640 10 monclient: dump: 2024-03-29T11:25:21.824 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:25:21.824 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:25:21.824 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:21.824 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:21.824 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:21.824 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:21.824 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:25:21.824 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:25:21.824 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:25:21.824 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:21.824 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.832+0000 7f45aa7fc640 1 -- 172.21.15.17:0/1268483716 <== mon.1 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f459c01a9d0 con 0x7f45ac0a9750 2024-03-29T11:25:21.824 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.833+0000 7f45b133e640 10 monclient: get_auth_request con 0x7f459806ddb0 auth_method 0 2024-03-29T11:25:21.825 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.833+0000 7f45aa7fc640 1 -- 172.21.15.17:0/1268483716 <== mon.1 v1:172.21.15.94:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f459c084970 con 0x7f45ac0a9750 2024-03-29T11:25:21.825 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.833+0000 7f45aa7fc640 10 client.4483.objecter ms_dispatch 0x7f45ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:21.825 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.833+0000 7f45aa7fc640 3 client.4483.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:21.825 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.833+0000 7f45aa7fc640 3 client.4483.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:21.825 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.833+0000 7f45aa7fc640 20 client.4483.objecter dump_active .. 0 homeless 2024-03-29T11:25:21.825 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.833+0000 7f45aa7fc640 10 client.4483.objecter ms_handle_connect 0x7f459806ddb0 2024-03-29T11:25:21.825 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.840+0000 7f45aa7fc640 1 -- 172.21.15.17:0/1268483716 <== mon.1 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f459c050250 con 0x7f45ac0a9750 2024-03-29T11:25:21.825 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.840+0000 7f45aa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:21.825 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:21.840+0000 7f45aa7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:22.198 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:22.214+0000 7f45b3da3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:22.199 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:22.214+0000 7f45b3da3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:22.199 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:22.214+0000 7f45b3da3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:25:22.199 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:22.214+0000 7f45b3da3640 1 -- 172.21.15.17:0/1268483716 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f458c005330 con 0x7f45ac0a9750 2024-03-29T11:25:22.199 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:22.216+0000 7f45aa7fc640 1 -- 172.21.15.17:0/1268483716 <== mon.1 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f459c052180 con 0x7f45ac0a9750 2024-03-29T11:25:22.199 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:22.216+0000 7f45aa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:22.200 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:22.216+0000 7f45aa7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:22.200 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:25:22.200 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:22.203 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:22.219+0000 7f45b3da3640 10 librados: watch_flush enter 2024-03-29T11:25:22.203 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:22.220+0000 7f45b3da3640 10 librados: watch_flush exit 2024-03-29T11:25:22.204 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:22.220+0000 7f45b3da3640 20 client.4483.objecter shutdown clearing up homeless session... 2024-03-29T11:25:22.204 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:22.220+0000 7f45b3da3640 10 client.4483.objecter successfully canceled tick 2024-03-29T11:25:22.204 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:22.220+0000 7f45b3da3640 1 -- 172.21.15.17:0/1268483716 >> v1:172.21.15.17:6816/3236185052 conn(0x7f459806ddb0 legacy=0x7f4598070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:22.204 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:22.220+0000 7f45b3da3640 10 monclient: shutdown 2024-03-29T11:25:22.204 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:22.220+0000 7f45b3da3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:22.204 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:22.220+0000 7f45b3da3640 1 -- 172.21.15.17:0/1268483716 >> v1:172.21.15.94:6789/0 conn(0x7f45ac0a9750 legacy=0x7f45ac13a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:22.205 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:22.220+0000 7f45b3da3640 1 -- 172.21.15.17:0/1268483716 shutdown_connections 2024-03-29T11:25:22.205 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:22.220+0000 7f45b3da3640 1 -- 172.21.15.17:0/1268483716 >> 172.21.15.17:0/1268483716 conn(0x7f45ac0b35e0 msgr2=0x7f45ac0b2500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:22.205 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:22.220+0000 7f45b3da3640 1 -- 172.21.15.17:0/1268483716 shutdown_connections 2024-03-29T11:25:22.205 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:22.221+0000 7f45b3da3640 1 -- 172.21.15.17:0/1268483716 wait complete. 2024-03-29T11:25:22.205 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:22.221+0000 7f45b3da3640 1 librados: shutdown 2024-03-29T11:25:22.223 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:29.030 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:29.178 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.194+0000 7f69ecd6d640 10 monclient: get_monmap_and_config 2024-03-29T11:25:29.178 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.194+0000 7f69ecd6d640 10 monclient: build_initial_monmap 2024-03-29T11:25:29.178 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.194+0000 7f69ecd6d640 10 monclient: monmap: 2024-03-29T11:25:29.178 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:25:29.178 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:25:29.178 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:25:29.195190+0000 2024-03-29T11:25:29.179 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:25:29.195190+0000 2024-03-29T11:25:29.179 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:29.179 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:29.179 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:25:29.179 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:25:29.179 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:25:29.179 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:29.179 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.195+0000 7f69ecd6d640 1 Processor -- start 2024-03-29T11:25:29.179 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.195+0000 7f69ecd6d640 1 -- start start 2024-03-29T11:25:29.180 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.195+0000 7f69ecd6d640 10 monclient: init 2024-03-29T11:25:29.183 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.196+0000 7f69ecd6d640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:29.184 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.196+0000 7f69ecd6d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:25:29.184 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.196+0000 7f69ecd6d640 10 monclient(hunting): picked mon.noname-c con 0x7f69e00389e0 addr v1:172.21.15.94:6789/0 2024-03-29T11:25:29.184 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.196+0000 7f69ecd6d640 10 monclient(hunting): picked mon.noname-a con 0x7f69e0038dc0 addr v1:172.21.15.17:6789/0 2024-03-29T11:25:29.184 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.196+0000 7f69ecd6d640 10 monclient(hunting): picked mon.noname-b con 0x7f69e0030660 addr v1:172.21.15.17:6790/0 2024-03-29T11:25:29.184 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.196+0000 7f69ecd6d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69e00c54c0 con 0x7f69e0038dc0 2024-03-29T11:25:29.184 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.196+0000 7f69ecd6d640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69e0036430 con 0x7f69e00389e0 2024-03-29T11:25:29.184 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.196+0000 7f69ecd6d640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69e0039840 con 0x7f69e0030660 2024-03-29T11:25:29.184 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.196+0000 7f69ecd6d640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:29.184 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.196+0000 7f69ecd6d640 10 monclient(hunting): authenticate will time out at 1235.401626s 2024-03-29T11:25:29.184 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.196+0000 7f69e77fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f69e00389e0 0x7f69e0033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:60574/0 (socket says 172.21.15.17:60574) 2024-03-29T11:25:29.184 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.196+0000 7f69e77fe640 1 -- 172.21.15.17:0/3720839844 learned_addr learned my addr 172.21.15.17:0/3720839844 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:25:29.184 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.197+0000 7f69e67fc640 1 -- 172.21.15.17:0/3720839844 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1073314972 0 0) 0x7f69e0036430 con 0x7f69e00389e0 2024-03-29T11:25:29.184 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.197+0000 7f69e67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:29.184 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.197+0000 7f69e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:29.185 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.197+0000 7f69e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:29.185 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.197+0000 7f69e67fc640 10 monclient(hunting): my global_id is 4486 2024-03-29T11:25:29.185 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.197+0000 7f69e67fc640 1 -- 172.21.15.17:0/3720839844 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69d0003520 con 0x7f69e00389e0 2024-03-29T11:25:29.185 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.197+0000 7f69e67fc640 1 -- 172.21.15.17:0/3720839844 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 504519543 0 0) 0x7f69e0039840 con 0x7f69e0030660 2024-03-29T11:25:29.185 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.197+0000 7f69e67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:29.185 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.197+0000 7f69e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:29.185 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.197+0000 7f69e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:29.185 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.197+0000 7f69e67fc640 10 monclient(hunting): my global_id is 4493 2024-03-29T11:25:29.185 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.197+0000 7f69e67fc640 1 -- 172.21.15.17:0/3720839844 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69e0036430 con 0x7f69e0030660 2024-03-29T11:25:29.185 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.197+0000 7f69e67fc640 1 -- 172.21.15.17:0/3720839844 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3405402402 0 0) 0x7f69d0003520 con 0x7f69e00389e0 2024-03-29T11:25:29.185 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.197+0000 7f69e67fc640 1 -- 172.21.15.17:0/3720839844 >> v1:172.21.15.17:6790/0 conn(0x7f69e0030660 legacy=0x7f69e00d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:29.185 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.197+0000 7f69e67fc640 1 -- 172.21.15.17:0/3720839844 >> v1:172.21.15.17:6789/0 conn(0x7f69e0038dc0 legacy=0x7f69e00cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:29.185 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.197+0000 7f69e67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:25:29.185 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.197+0000 7f69e67fc640 1 monclient: found mon.noname-c 2024-03-29T11:25:29.186 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.197+0000 7f69e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:29.186 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.197+0000 7f69e67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6789/0 2024-03-29T11:25:29.186 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.197+0000 7f69e67fc640 1 -- 172.21.15.17:0/3720839844 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e0035690 con 0x7f69e00389e0 2024-03-29T11:25:29.186 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.197+0000 7f69e67fc640 10 monclient: _finish_auth 0 2024-03-29T11:25:29.186 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.197+0000 7f69e67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:25:29.186 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.197+0000 7f69e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:29.186 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.197+0000 7f69ecd6d640 5 monclient: authenticate success, global_id 4486 2024-03-29T11:25:29.186 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.197+0000 7f69ecd6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:29.186 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.197+0000 7f69e67fc640 1 -- 172.21.15.17:0/3720839844 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f69dc001690 con 0x7f69e00389e0 2024-03-29T11:25:29.186 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.197+0000 7f69e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:29.186 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.197+0000 7f69e67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:29.186 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.197+0000 7f69e67fc640 10 monclient: dump: 2024-03-29T11:25:29.186 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:25:29.186 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:25:29.187 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:29.187 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:29.187 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:29.187 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:29.187 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:25:29.187 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:25:29.187 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:25:29.187 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:29.187 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.197+0000 7f69ecd6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:29.187 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.198+0000 7f69e67fc640 1 -- 172.21.15.17:0/3720839844 <== mon.1 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69dc001960 con 0x7f69e00389e0 2024-03-29T11:25:29.187 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.198+0000 7f69e67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:29.187 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.198+0000 7f69e67fc640 1 -- 172.21.15.17:0/3720839844 <== mon.1 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f69dc001cf0 con 0x7f69e00389e0 2024-03-29T11:25:29.187 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.198+0000 7f69e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:29.187 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.198+0000 7f69e67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:29.187 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.198+0000 7f69e67fc640 10 monclient: dump: 2024-03-29T11:25:29.188 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:25:29.188 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:25:29.188 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:29.188 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:29.188 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:29.188 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:29.188 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:25:29.188 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:25:29.188 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:25:29.188 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:29.188 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.198+0000 7f69ecd6d640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:29.188 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.198+0000 7f69ecd6d640 4 set_mon_vals no callback set 2024-03-29T11:25:29.188 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.198+0000 7f69ecd6d640 10 monclient: shutdown 2024-03-29T11:25:29.188 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.198+0000 7f69ecd6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:29.188 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.198+0000 7f69ecd6d640 1 -- 172.21.15.17:0/3720839844 >> v1:172.21.15.94:6789/0 conn(0x7f69e00389e0 legacy=0x7f69e0033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:29.189 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.199+0000 7f69ecd6d640 1 -- 172.21.15.17:0/3720839844 shutdown_connections 2024-03-29T11:25:29.189 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.199+0000 7f69ecd6d640 1 -- 172.21.15.17:0/3720839844 >> 172.21.15.17:0/3720839844 conn(0x7f69e0005c00 msgr2=0x7f69e0006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:29.189 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.199+0000 7f69ecd6d640 1 -- 172.21.15.17:0/3720839844 shutdown_connections 2024-03-29T11:25:29.189 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.199+0000 7f69ecd6d640 1 -- 172.21.15.17:0/3720839844 wait complete. 2024-03-29T11:25:29.196 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.200+0000 7f69ecd6d640 10 monclient: build_initial_monmap 2024-03-29T11:25:29.197 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.200+0000 7f69ecd6d640 10 monclient: monmap: 2024-03-29T11:25:29.197 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:25:29.197 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:29.197 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:25:29.197 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:25:29.197 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:29.197 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:29.197 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:25:29.197 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:25:29.197 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:25:29.197 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:29.197 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.201+0000 7f69ecd6d640 1 librados: starting msgr at 2024-03-29T11:25:29.197 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.201+0000 7f69ecd6d640 1 librados: starting objecter 2024-03-29T11:25:29.197 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.201+0000 7f69ecd6d640 1 Processor -- start 2024-03-29T11:25:29.197 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.202+0000 7f69ecd6d640 1 -- start start 2024-03-29T11:25:29.198 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.202+0000 7f69ecd6d640 1 librados: setting wanted keys 2024-03-29T11:25:29.198 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.202+0000 7f69ecd6d640 1 librados: calling monclient init 2024-03-29T11:25:29.198 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.202+0000 7f69ecd6d640 10 monclient: init 2024-03-29T11:25:29.198 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.202+0000 7f69ecd6d640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:29.198 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.202+0000 7f69ecd6d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:25:29.198 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.202+0000 7f69ecd6d640 10 monclient(hunting): picked mon.noname-b con 0x7f69e0030660 addr v1:172.21.15.94:6789/0 2024-03-29T11:25:29.198 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.202+0000 7f69ecd6d640 10 monclient(hunting): picked mon.noname-a con 0x7f69e00389e0 addr v1:172.21.15.17:6789/0 2024-03-29T11:25:29.198 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.202+0000 7f69ecd6d640 10 monclient(hunting): picked mon.noname-c con 0x7f69e00c7500 addr v1:172.21.15.17:6790/0 2024-03-29T11:25:29.198 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.202+0000 7f69ecd6d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69e002f7d0 con 0x7f69e00389e0 2024-03-29T11:25:29.198 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.202+0000 7f69e77fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f69e0030660 0x7f69e002d150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:32978/0 (socket says 172.21.15.17:32978) 2024-03-29T11:25:29.198 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.202+0000 7f69e77fe640 1 -- 172.21.15.17:0/1777380613 learned_addr learned my addr 172.21.15.17:0/1777380613 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:25:29.198 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.202+0000 7f69ecd6d640 1 -- 172.21.15.17:0/1777380613 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69e0009d30 con 0x7f69e0030660 2024-03-29T11:25:29.198 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.203+0000 7f69ecd6d640 1 -- 172.21.15.17:0/1777380613 --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69e00c2bd0 con 0x7f69e00c7500 2024-03-29T11:25:29.198 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.203+0000 7f69ecd6d640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:29.199 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.203+0000 7f69ecd6d640 10 monclient(hunting): authenticate will time out at 1235.408091s 2024-03-29T11:25:29.199 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.203+0000 7f69cffff640 10 client.?.objecter ms_handle_connect 0x7f69e0030660 2024-03-29T11:25:29.199 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.203+0000 7f69cffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:29.199 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.203+0000 7f69cffff640 1 -- 172.21.15.17:0/1777380613 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 169459894 0 0) 0x7f69e0009d30 con 0x7f69e0030660 2024-03-29T11:25:29.199 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.203+0000 7f69cffff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:29.199 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.203+0000 7f69cffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:29.199 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.203+0000 7f69cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:29.199 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.203+0000 7f69cffff640 10 monclient(hunting): my global_id is 4489 2024-03-29T11:25:29.199 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.203+0000 7f69cffff640 1 -- 172.21.15.17:0/1777380613 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69d4004eb0 con 0x7f69e0030660 2024-03-29T11:25:29.199 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.203+0000 7f69cffff640 10 client.?.objecter ms_handle_connect 0x7f69e00c7500 2024-03-29T11:25:29.199 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.203+0000 7f69cffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:29.199 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.203+0000 7f69cffff640 1 -- 172.21.15.17:0/1777380613 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 331529367 0 0) 0x7f69d4004eb0 con 0x7f69e0030660 2024-03-29T11:25:29.199 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.204+0000 7f69cffff640 1 -- 172.21.15.17:0/1777380613 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f69e0009d30 con 0x7f69e0030660 2024-03-29T11:25:29.199 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.204+0000 7f69cffff640 10 client.?.objecter ms_handle_connect 0x7f69e00389e0 2024-03-29T11:25:29.199 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.204+0000 7f69cffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:29.200 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.204+0000 7f69cffff640 1 -- 172.21.15.17:0/1777380613 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f69dc001650 con 0x7f69e0030660 2024-03-29T11:25:29.200 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.204+0000 7f69cffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:29.200 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.204+0000 7f69cffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:29.200 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.204+0000 7f69cffff640 10 monclient(hunting): dump: 2024-03-29T11:25:29.200 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:25:29.200 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:25:29.200 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:29.200 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:29.200 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:29.200 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:29.200 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:25:29.200 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:25:29.200 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:25:29.200 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:29.201 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.204+0000 7f69cffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:29.201 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.204+0000 7f69cffff640 1 -- 172.21.15.17:0/1777380613 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3584267667 0 0) 0x7f69e002f7d0 con 0x7f69e00389e0 2024-03-29T11:25:29.201 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.204+0000 7f69cffff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:29.201 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.204+0000 7f69cffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:29.201 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.204+0000 7f69cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:29.201 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.204+0000 7f69cffff640 10 monclient(hunting): my global_id is 4449 2024-03-29T11:25:29.201 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.204+0000 7f69cffff640 1 -- 172.21.15.17:0/1777380613 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69d4004eb0 con 0x7f69e00389e0 2024-03-29T11:25:29.201 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.204+0000 7f69cffff640 1 -- 172.21.15.17:0/1777380613 <== mon.1 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4278545253 0 0) 0x7f69e0009d30 con 0x7f69e0030660 2024-03-29T11:25:29.201 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.204+0000 7f69cffff640 1 -- 172.21.15.17:0/1777380613 >> v1:172.21.15.17:6790/0 conn(0x7f69e00c7500 legacy=0x7f69e016b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:29.201 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.204+0000 7f69cffff640 1 -- 172.21.15.17:0/1777380613 >> v1:172.21.15.17:6789/0 conn(0x7f69e00389e0 legacy=0x7f69e002d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:29.201 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.204+0000 7f69cffff640 10 monclient: _finish_hunting 0 2024-03-29T11:25:29.201 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.204+0000 7f69cffff640 1 monclient: found mon.b 2024-03-29T11:25:29.201 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.204+0000 7f69cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:29.201 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.204+0000 7f69cffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:25:29.201 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.204+0000 7f69cffff640 1 -- 172.21.15.17:0/1777380613 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e00c2d60 con 0x7f69e0030660 2024-03-29T11:25:29.202 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.204+0000 7f69cffff640 10 monclient: _finish_auth 0 2024-03-29T11:25:29.202 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.204+0000 7f69cffff640 10 monclient: _check_auth_tickets 2024-03-29T11:25:29.202 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.204+0000 7f69cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:29.202 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.204+0000 7f69ecd6d640 5 monclient: authenticate success, global_id 4489 2024-03-29T11:25:29.202 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.204+0000 7f69ecd6d640 10 monclient: _renew_subs 2024-03-29T11:25:29.202 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.204+0000 7f69ecd6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:25:29.202 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.204+0000 7f69ecd6d640 1 -- 172.21.15.17:0/1777380613 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69e002f0b0 con 0x7f69e0030660 2024-03-29T11:25:29.202 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.204+0000 7f69ecd6d640 10 client.4489.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:29.202 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.204+0000 7f69ecd6d640 10 monclient: _renew_subs 2024-03-29T11:25:29.202 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.204+0000 7f69ecd6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:25:29.202 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.204+0000 7f69ecd6d640 1 -- 172.21.15.17:0/1777380613 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f69e016ceb0 con 0x7f69e0030660 2024-03-29T11:25:29.202 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.204+0000 7f69ecd6d640 1 librados: init done 2024-03-29T11:25:29.202 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.205+0000 7f69cffff640 1 -- 172.21.15.17:0/1777380613 <== mon.1 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69dc004ca0 con 0x7f69e0030660 2024-03-29T11:25:29.202 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.205+0000 7f69cffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:29.203 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.205+0000 7f69cffff640 1 -- 172.21.15.17:0/1777380613 <== mon.1 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f69dc0050b0 con 0x7f69e0030660 2024-03-29T11:25:29.203 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.205+0000 7f69cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:29.203 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.205+0000 7f69cffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:29.203 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.205+0000 7f69e5ffb640 4 set_mon_vals no callback set 2024-03-29T11:25:29.203 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.205+0000 7f69cffff640 10 monclient: dump: 2024-03-29T11:25:29.203 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:25:29.203 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:25:29.203 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:29.203 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:29.203 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:29.203 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:29.203 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:25:29.203 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:25:29.203 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:25:29.203 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:29.204 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.206+0000 7f69ecd6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:29.204 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.206+0000 7f69ecd6d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:29.204 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.206+0000 7f69ecd6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:25:29.204 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.206+0000 7f69ecd6d640 1 -- 172.21.15.17:0/1777380613 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69c8005080 con 0x7f69e0030660 2024-03-29T11:25:29.204 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.207+0000 7f69cffff640 1 -- 172.21.15.17:0/1777380613 <== mon.1 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f69dc005210 con 0x7f69e0030660 2024-03-29T11:25:29.204 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.208+0000 7f69e6ffd640 10 monclient: get_auth_request con 0x7f69d406dd70 auth_method 0 2024-03-29T11:25:29.204 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.208+0000 7f69cffff640 10 client.4489.objecter ms_handle_connect 0x7f69d406dd70 2024-03-29T11:25:29.204 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.208+0000 7f69cffff640 1 -- 172.21.15.17:0/1777380613 <== mon.1 v1:172.21.15.94:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f69dc084cc0 con 0x7f69e0030660 2024-03-29T11:25:29.204 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.208+0000 7f69cffff640 10 client.4489.objecter ms_dispatch 0x7f69e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:29.204 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.208+0000 7f69cffff640 3 client.4489.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:29.204 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.208+0000 7f69cffff640 3 client.4489.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:29.204 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.208+0000 7f69cffff640 20 client.4489.objecter dump_active .. 0 homeless 2024-03-29T11:25:29.205 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.216+0000 7f69cffff640 1 -- 172.21.15.17:0/1777380613 <== mon.1 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f69dc0505a0 con 0x7f69e0030660 2024-03-29T11:25:29.205 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.216+0000 7f69cffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:29.205 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.216+0000 7f69cffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:29.626 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.642+0000 7f69ecd6d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:29.626 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.642+0000 7f69ecd6d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:29.626 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.642+0000 7f69ecd6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:25:29.626 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.642+0000 7f69ecd6d640 1 -- 172.21.15.17:0/1777380613 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f69c8005330 con 0x7f69e0030660 2024-03-29T11:25:29.627 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.643+0000 7f69cffff640 1 -- 172.21.15.17:0/1777380613 <== mon.1 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f69dc0524d0 con 0x7f69e0030660 2024-03-29T11:25:29.627 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.643+0000 7f69cffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:29.627 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.643+0000 7f69cffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:29.627 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:25:29.627 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:29.630 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.646+0000 7f69ecd6d640 10 librados: watch_flush enter 2024-03-29T11:25:29.631 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.646+0000 7f69ecd6d640 10 librados: watch_flush exit 2024-03-29T11:25:29.631 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.646+0000 7f69ecd6d640 20 client.4489.objecter shutdown clearing up homeless session... 2024-03-29T11:25:29.631 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.646+0000 7f69ecd6d640 10 client.4489.objecter successfully canceled tick 2024-03-29T11:25:29.631 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.646+0000 7f69ecd6d640 1 -- 172.21.15.17:0/1777380613 >> v1:172.21.15.17:6816/3236185052 conn(0x7f69d406dd70 legacy=0x7f69d4070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:29.631 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.646+0000 7f69ecd6d640 10 monclient: shutdown 2024-03-29T11:25:29.631 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.646+0000 7f69ecd6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:29.631 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.646+0000 7f69ecd6d640 1 -- 172.21.15.17:0/1777380613 >> v1:172.21.15.94:6789/0 conn(0x7f69e0030660 legacy=0x7f69e002d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:29.631 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.646+0000 7f69ecd6d640 1 -- 172.21.15.17:0/1777380613 shutdown_connections 2024-03-29T11:25:29.631 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.646+0000 7f69ecd6d640 1 -- 172.21.15.17:0/1777380613 >> 172.21.15.17:0/1777380613 conn(0x7f69e0005c00 msgr2=0x7f69e0005fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:29.632 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.647+0000 7f69ecd6d640 1 -- 172.21.15.17:0/1777380613 shutdown_connections 2024-03-29T11:25:29.632 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.647+0000 7f69ecd6d640 1 -- 172.21.15.17:0/1777380613 wait complete. 2024-03-29T11:25:29.632 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:29.647+0000 7f69ecd6d640 1 librados: shutdown 2024-03-29T11:25:29.647 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:36.454 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:36.603 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.618+0000 7f4f6bfff640 10 monclient: get_monmap_and_config 2024-03-29T11:25:36.603 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.618+0000 7f4f6bfff640 10 monclient: build_initial_monmap 2024-03-29T11:25:36.603 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.618+0000 7f4f6bfff640 10 monclient: monmap: 2024-03-29T11:25:36.603 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:25:36.603 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:25:36.603 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:25:36.619422+0000 2024-03-29T11:25:36.603 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:25:36.619422+0000 2024-03-29T11:25:36.603 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:36.603 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:36.603 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:25:36.603 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:25:36.603 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:25:36.604 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:36.604 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.619+0000 7f4f6bfff640 1 Processor -- start 2024-03-29T11:25:36.604 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.619+0000 7f4f6bfff640 1 -- start start 2024-03-29T11:25:36.604 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.619+0000 7f4f6bfff640 10 monclient: init 2024-03-29T11:25:36.606 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.620+0000 7f4f6bfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:36.606 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.620+0000 7f4f6bfff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:25:36.607 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.620+0000 7f4f6bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f4f6c0b4920 addr v1:172.21.15.17:6790/0 2024-03-29T11:25:36.607 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.620+0000 7f4f6bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f4f6c0b4200 addr v1:172.21.15.94:6789/0 2024-03-29T11:25:36.607 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.620+0000 7f4f6bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f4f6c13f9a0 addr v1:172.21.15.17:6789/0 2024-03-29T11:25:36.607 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.620+0000 7f4f6bfff640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f6c0b0930 con 0x7f4f6c13f9a0 2024-03-29T11:25:36.607 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.620+0000 7f4f6bfff640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f6c0af460 con 0x7f4f6c0b4200 2024-03-29T11:25:36.607 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.620+0000 7f4f6bfff640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f6c0aeda0 con 0x7f4f6c0b4920 2024-03-29T11:25:36.607 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.620+0000 7f4f6bfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:36.607 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.620+0000 7f4f6bfff640 10 monclient(hunting): authenticate will time out at 1242.825234s 2024-03-29T11:25:36.608 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.622+0000 7f4f6affd640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f4f6c0b4920 0x7f4f6c13f290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:58466/0 (socket says 172.21.15.17:58466) 2024-03-29T11:25:36.608 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.623+0000 7f4f6affd640 1 -- 172.21.15.17:0/3873289934 learned_addr learned my addr 172.21.15.17:0/3873289934 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:25:36.608 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.623+0000 7f4f69ffb640 1 -- 172.21.15.17:0/3873289934 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4190914640 0 0) 0x7f4f6c0b0930 con 0x7f4f6c13f9a0 2024-03-29T11:25:36.608 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.623+0000 7f4f69ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:36.608 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.623+0000 7f4f69ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:36.608 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.623+0000 7f4f69ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:36.608 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.623+0000 7f4f69ffb640 10 monclient(hunting): my global_id is 4452 2024-03-29T11:25:36.608 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.623+0000 7f4f69ffb640 1 -- 172.21.15.17:0/3873289934 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f54003520 con 0x7f4f6c13f9a0 2024-03-29T11:25:36.608 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.624+0000 7f4f69ffb640 1 -- 172.21.15.17:0/3873289934 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1953769860 0 0) 0x7f4f54003520 con 0x7f4f6c13f9a0 2024-03-29T11:25:36.608 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.624+0000 7f4f69ffb640 1 -- 172.21.15.17:0/3873289934 >> v1:172.21.15.17:6790/0 conn(0x7f4f6c0b4920 legacy=0x7f4f6c13f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:36.609 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.624+0000 7f4f69ffb640 1 -- 172.21.15.17:0/3873289934 >> v1:172.21.15.94:6789/0 conn(0x7f4f6c0b4200 legacy=0x7f4f6c0b0fc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:36.609 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.624+0000 7f4f69ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:25:36.609 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.624+0000 7f4f69ffb640 1 monclient: found mon.noname-a 2024-03-29T11:25:36.609 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.624+0000 7f4f69ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:36.609 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.624+0000 7f4f69ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.17:6789/0 2024-03-29T11:25:36.609 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.624+0000 7f4f69ffb640 1 -- 172.21.15.17:0/3873289934 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f6c0af980 con 0x7f4f6c13f9a0 2024-03-29T11:25:36.609 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.624+0000 7f4f69ffb640 10 monclient: _finish_auth 0 2024-03-29T11:25:36.609 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.624+0000 7f4f69ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:25:36.610 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.624+0000 7f4f69ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:36.610 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.624+0000 7f4f69ffb640 1 -- 172.21.15.17:0/3873289934 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f4f64001a10 con 0x7f4f6c13f9a0 2024-03-29T11:25:36.610 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.624+0000 7f4f69ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:36.610 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.624+0000 7f4f69ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:36.610 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.624+0000 7f4f69ffb640 10 monclient: dump: 2024-03-29T11:25:36.610 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:25:36.610 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:25:36.610 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:36.610 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:36.610 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:36.610 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:36.610 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:25:36.610 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:25:36.610 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:25:36.611 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:36.611 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.624+0000 7f4f6bfff640 5 monclient: authenticate success, global_id 4452 2024-03-29T11:25:36.611 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.624+0000 7f4f6bfff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:36.612 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.626+0000 7f4f69ffb640 1 -- 172.21.15.17:0/3873289934 <== mon.0 v1:172.21.15.17:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f64001be0 con 0x7f4f6c13f9a0 2024-03-29T11:25:36.612 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.626+0000 7f4f69ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:36.612 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.626+0000 7f4f69ffb640 1 -- 172.21.15.17:0/3873289934 <== mon.0 v1:172.21.15.17:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f4f64002030 con 0x7f4f6c13f9a0 2024-03-29T11:25:36.612 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.626+0000 7f4f69ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:36.612 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.626+0000 7f4f69ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:36.612 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.626+0000 7f4f69ffb640 10 monclient: dump: 2024-03-29T11:25:36.612 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:25:36.612 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:25:36.612 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:36.612 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:36.612 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:36.613 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:36.613 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:25:36.613 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:25:36.613 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:25:36.613 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:36.613 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.626+0000 7f4f6bfff640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:36.613 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.626+0000 7f4f6bfff640 4 set_mon_vals no callback set 2024-03-29T11:25:36.613 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.627+0000 7f4f6bfff640 10 monclient: shutdown 2024-03-29T11:25:36.613 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.627+0000 7f4f6bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:36.613 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.627+0000 7f4f6bfff640 1 -- 172.21.15.17:0/3873289934 >> v1:172.21.15.17:6789/0 conn(0x7f4f6c13f9a0 legacy=0x7f4f6c14d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:36.613 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.627+0000 7f4f6bfff640 1 -- 172.21.15.17:0/3873289934 shutdown_connections 2024-03-29T11:25:36.613 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.627+0000 7f4f6bfff640 1 -- 172.21.15.17:0/3873289934 >> 172.21.15.17:0/3873289934 conn(0x7f4f6c06d230 msgr2=0x7f4f6c066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:36.613 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.627+0000 7f4f6bfff640 1 -- 172.21.15.17:0/3873289934 shutdown_connections 2024-03-29T11:25:36.613 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.627+0000 7f4f6bfff640 1 -- 172.21.15.17:0/3873289934 wait complete. 2024-03-29T11:25:36.614 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.628+0000 7f4f6bfff640 10 monclient: build_initial_monmap 2024-03-29T11:25:36.615 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.628+0000 7f4f6bfff640 10 monclient: monmap: 2024-03-29T11:25:36.615 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:25:36.615 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:36.615 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:25:36.615 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:25:36.615 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:36.615 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:36.615 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:25:36.615 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:25:36.615 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:25:36.615 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:36.615 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.628+0000 7f4f6bfff640 1 librados: starting msgr at 2024-03-29T11:25:36.615 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.628+0000 7f4f6bfff640 1 librados: starting objecter 2024-03-29T11:25:36.615 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.628+0000 7f4f6bfff640 1 Processor -- start 2024-03-29T11:25:36.616 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.629+0000 7f4f6bfff640 1 -- start start 2024-03-29T11:25:36.616 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.629+0000 7f4f6bfff640 1 librados: setting wanted keys 2024-03-29T11:25:36.616 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.629+0000 7f4f6bfff640 1 librados: calling monclient init 2024-03-29T11:25:36.619 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.629+0000 7f4f6bfff640 10 monclient: init 2024-03-29T11:25:36.619 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.629+0000 7f4f6bfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:36.620 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.629+0000 7f4f6bfff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:25:36.620 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.630+0000 7f4f6bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f4f6c0b4200 addr v1:172.21.15.17:6790/0 2024-03-29T11:25:36.620 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.630+0000 7f4f6bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f4f6c0b4920 addr v1:172.21.15.94:6789/0 2024-03-29T11:25:36.620 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.630+0000 7f4f6bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f4f6c13f9a0 addr v1:172.21.15.17:6789/0 2024-03-29T11:25:36.620 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.630+0000 7f4f6bfff640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f6c0ae5a0 con 0x7f4f6c13f9a0 2024-03-29T11:25:36.620 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.630+0000 7f4f6bfff640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f6c1408a0 con 0x7f4f6c0b4920 2024-03-29T11:25:36.620 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.630+0000 7f4f6bfff640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f6c140b80 con 0x7f4f6c0b4200 2024-03-29T11:25:36.620 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.630+0000 7f4f6bfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:36.620 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.630+0000 7f4f6bfff640 10 monclient(hunting): authenticate will time out at 1242.834957s 2024-03-29T11:25:36.620 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.633+0000 7f4f6b7fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f4f6c13f9a0 0x7f4f6c1e5c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:47279/0 (socket says 172.21.15.17:47279) 2024-03-29T11:25:36.620 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.633+0000 7f4f6b7fe640 1 -- 172.21.15.17:0/955982047 learned_addr learned my addr 172.21.15.17:0/955982047 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:25:36.620 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.633+0000 7f4f53fff640 10 client.?.objecter ms_handle_connect 0x7f4f6c0b4200 2024-03-29T11:25:36.620 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.633+0000 7f4f53fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:36.620 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.633+0000 7f4f53fff640 10 client.?.objecter ms_handle_connect 0x7f4f6c13f9a0 2024-03-29T11:25:36.620 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.633+0000 7f4f53fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:36.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.633+0000 7f4f53fff640 1 -- 172.21.15.17:0/955982047 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 691660800 0 0) 0x7f4f6c140b80 con 0x7f4f6c0b4200 2024-03-29T11:25:36.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.633+0000 7f4f53fff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:36.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.633+0000 7f4f53fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:36.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.633+0000 7f4f53fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:36.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.633+0000 7f4f53fff640 10 monclient(hunting): my global_id is 4499 2024-03-29T11:25:36.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.633+0000 7f4f53fff640 1 -- 172.21.15.17:0/955982047 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f58015ec0 con 0x7f4f6c0b4200 2024-03-29T11:25:36.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.633+0000 7f4f53fff640 1 -- 172.21.15.17:0/955982047 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1423208960 0 0) 0x7f4f6c0ae5a0 con 0x7f4f6c13f9a0 2024-03-29T11:25:36.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.633+0000 7f4f53fff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:36.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.633+0000 7f4f53fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:36.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.633+0000 7f4f53fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:36.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.633+0000 7f4f53fff640 10 monclient(hunting): my global_id is 4455 2024-03-29T11:25:36.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.633+0000 7f4f53fff640 1 -- 172.21.15.17:0/955982047 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f6c140b80 con 0x7f4f6c13f9a0 2024-03-29T11:25:36.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.634+0000 7f4f53fff640 1 -- 172.21.15.17:0/955982047 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 312882605 0 0) 0x7f4f58015ec0 con 0x7f4f6c0b4200 2024-03-29T11:25:36.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.634+0000 7f4f53fff640 1 -- 172.21.15.17:0/955982047 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f580164a0 con 0x7f4f6c0b4200 2024-03-29T11:25:36.621 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.634+0000 7f4f53fff640 1 -- 172.21.15.17:0/955982047 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f4f5c0018a0 con 0x7f4f6c0b4200 2024-03-29T11:25:36.622 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.634+0000 7f4f53fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:36.622 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.634+0000 7f4f53fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:36.622 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.634+0000 7f4f53fff640 10 monclient(hunting): dump: 2024-03-29T11:25:36.622 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:25:36.622 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:25:36.622 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:36.622 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:36.622 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:36.622 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:36.622 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:25:36.622 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:25:36.622 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:25:36.622 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:36.622 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.634+0000 7f4f53fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:36.622 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.634+0000 7f4f53fff640 1 -- 172.21.15.17:0/955982047 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2972456828 0 0) 0x7f4f6c140b80 con 0x7f4f6c13f9a0 2024-03-29T11:25:36.623 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.634+0000 7f4f53fff640 1 -- 172.21.15.17:0/955982047 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f58015ec0 con 0x7f4f6c13f9a0 2024-03-29T11:25:36.623 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.634+0000 7f4f53fff640 1 -- 172.21.15.17:0/955982047 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f4f64001ef0 con 0x7f4f6c13f9a0 2024-03-29T11:25:36.623 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.634+0000 7f4f53fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:36.623 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.634+0000 7f4f53fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:36.623 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.634+0000 7f4f53fff640 10 monclient(hunting): dump: 2024-03-29T11:25:36.623 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:25:36.623 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:25:36.623 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:36.623 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:36.623 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:36.623 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:36.623 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:25:36.623 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:25:36.623 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:25:36.623 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:36.623 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.635+0000 7f4f53fff640 1 -- 172.21.15.17:0/955982047 <== mon.2 v1:172.21.15.17:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2805303870 0 0) 0x7f4f580164a0 con 0x7f4f6c0b4200 2024-03-29T11:25:36.624 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.635+0000 7f4f53fff640 1 -- 172.21.15.17:0/955982047 >> v1:172.21.15.94:6789/0 conn(0x7f4f6c0b4920 legacy=0x7f4f6c1dbae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:25:36.627 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.636+0000 7f4f53fff640 1 -- 172.21.15.17:0/955982047 >> v1:172.21.15.17:6789/0 conn(0x7f4f6c13f9a0 legacy=0x7f4f6c1e5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:36.627 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.636+0000 7f4f53fff640 10 monclient: _finish_hunting 0 2024-03-29T11:25:36.627 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.636+0000 7f4f53fff640 1 monclient: found mon.c 2024-03-29T11:25:36.627 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.636+0000 7f4f53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:36.627 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.636+0000 7f4f53fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:25:36.627 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.636+0000 7f4f53fff640 1 -- 172.21.15.17:0/955982047 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f6c140e60 con 0x7f4f6c0b4200 2024-03-29T11:25:36.627 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.636+0000 7f4f53fff640 10 monclient: _finish_auth 0 2024-03-29T11:25:36.628 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.636+0000 7f4f53fff640 10 monclient: _check_auth_tickets 2024-03-29T11:25:36.628 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.636+0000 7f4f53fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:36.628 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.636+0000 7f4f6bfff640 5 monclient: authenticate success, global_id 4499 2024-03-29T11:25:36.628 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.636+0000 7f4f6bfff640 10 monclient: _renew_subs 2024-03-29T11:25:36.628 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.636+0000 7f4f6bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:25:36.628 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.636+0000 7f4f6bfff640 1 -- 172.21.15.17:0/955982047 --> v1:172.21.15.17:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f6c0ae230 con 0x7f4f6c0b4200 2024-03-29T11:25:36.628 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.636+0000 7f4f6bfff640 10 client.4499.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:36.628 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.636+0000 7f4f6bfff640 10 monclient: _renew_subs 2024-03-29T11:25:36.628 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.636+0000 7f4f6bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:25:36.628 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.636+0000 7f4f6bfff640 1 -- 172.21.15.17:0/955982047 --> v1:172.21.15.17:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f6c0ad820 con 0x7f4f6c0b4200 2024-03-29T11:25:36.628 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.636+0000 7f4f6bfff640 1 librados: init done 2024-03-29T11:25:36.628 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.636+0000 7f4f53fff640 1 -- 172.21.15.17:0/955982047 <== mon.2 v1:172.21.15.17:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f5c001d10 con 0x7f4f6c0b4200 2024-03-29T11:25:36.628 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.636+0000 7f4f53fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:36.628 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.636+0000 7f4f53fff640 1 -- 172.21.15.17:0/955982047 <== mon.2 v1:172.21.15.17:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f4f5c004980 con 0x7f4f6c0b4200 2024-03-29T11:25:36.628 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.636+0000 7f4f53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:36.628 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.636+0000 7f4f53fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:36.629 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.636+0000 7f4f53fff640 10 monclient: dump: 2024-03-29T11:25:36.629 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:25:36.629 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:25:36.629 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:36.629 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:36.629 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:36.629 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:36.629 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:25:36.629 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:25:36.629 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:25:36.629 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:36.629 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.636+0000 7f4f68ff9640 4 set_mon_vals no callback set 2024-03-29T11:25:36.629 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.638+0000 7f4f53fff640 1 -- 172.21.15.17:0/955982047 <== mon.2 v1:172.21.15.17:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f4f5c01a950 con 0x7f4f6c0b4200 2024-03-29T11:25:36.629 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.639+0000 7f4f53fff640 1 -- 172.21.15.17:0/955982047 <== mon.2 v1:172.21.15.17:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f4f5c084a20 con 0x7f4f6c0b4200 2024-03-29T11:25:36.629 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.639+0000 7f4f53fff640 10 client.4499.objecter ms_dispatch 0x7f4f6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:36.630 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.639+0000 7f4f53fff640 3 client.4499.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:36.630 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.639+0000 7f4f53fff640 3 client.4499.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:36.630 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.639+0000 7f4f53fff640 20 client.4499.objecter dump_active .. 0 homeless 2024-03-29T11:25:36.630 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.639+0000 7f4f6a7fc640 10 monclient: get_auth_request con 0x7f4f5807f200 auth_method 0 2024-03-29T11:25:36.630 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.640+0000 7f4f53fff640 10 client.4499.objecter ms_handle_connect 0x7f4f5807f200 2024-03-29T11:25:36.630 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.641+0000 7f4f6bfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:36.630 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.641+0000 7f4f6bfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:36.630 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.641+0000 7f4f6bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:25:36.630 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.641+0000 7f4f6bfff640 1 -- 172.21.15.17:0/955982047 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f4c005080 con 0x7f4f6c0b4200 2024-03-29T11:25:36.635 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.651+0000 7f4f53fff640 1 -- 172.21.15.17:0/955982047 <== mon.2 v1:172.21.15.17:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4f5c050130 con 0x7f4f6c0b4200 2024-03-29T11:25:36.635 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.651+0000 7f4f53fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:36.635 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.651+0000 7f4f53fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:36.978 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.993+0000 7f4f6bfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:36.978 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.993+0000 7f4f6bfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:36.978 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.993+0000 7f4f6bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:25:36.978 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.993+0000 7f4f6bfff640 1 -- 172.21.15.17:0/955982047 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f4c005330 con 0x7f4f6c0b4200 2024-03-29T11:25:36.979 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.994+0000 7f4f53fff640 1 -- 172.21.15.17:0/955982047 <== mon.2 v1:172.21.15.17:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f4f5c052230 con 0x7f4f6c0b4200 2024-03-29T11:25:36.979 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.994+0000 7f4f53fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:36.979 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.994+0000 7f4f53fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:36.979 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:25:36.979 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:36.982 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.998+0000 7f4f6bfff640 10 librados: watch_flush enter 2024-03-29T11:25:36.982 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.998+0000 7f4f6bfff640 10 librados: watch_flush exit 2024-03-29T11:25:36.983 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.998+0000 7f4f6bfff640 20 client.4499.objecter shutdown clearing up homeless session... 2024-03-29T11:25:36.983 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.998+0000 7f4f6bfff640 10 client.4499.objecter successfully canceled tick 2024-03-29T11:25:36.983 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.998+0000 7f4f6bfff640 1 -- 172.21.15.17:0/955982047 >> v1:172.21.15.17:6816/3236185052 conn(0x7f4f5807f200 legacy=0x7f4f580816c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:36.983 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.998+0000 7f4f6bfff640 10 monclient: shutdown 2024-03-29T11:25:36.983 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.998+0000 7f4f6bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:36.983 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.998+0000 7f4f6bfff640 1 -- 172.21.15.17:0/955982047 >> v1:172.21.15.17:6790/0 conn(0x7f4f6c0b4200 legacy=0x7f4f6c1db3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:36.983 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.998+0000 7f4f6bfff640 1 -- 172.21.15.17:0/955982047 shutdown_connections 2024-03-29T11:25:36.983 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.998+0000 7f4f6bfff640 1 -- 172.21.15.17:0/955982047 >> 172.21.15.17:0/955982047 conn(0x7f4f6c06d230 msgr2=0x7f4f6c0b6a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:36.983 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.998+0000 7f4f6bfff640 1 -- 172.21.15.17:0/955982047 shutdown_connections 2024-03-29T11:25:36.983 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.999+0000 7f4f6bfff640 1 -- 172.21.15.17:0/955982047 wait complete. 2024-03-29T11:25:36.984 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:36.999+0000 7f4f6bfff640 1 librados: shutdown 2024-03-29T11:25:36.998 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:43.806 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:43.955 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.970+0000 7fc82f59e640 10 monclient: get_monmap_and_config 2024-03-29T11:25:43.955 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.970+0000 7fc82f59e640 10 monclient: build_initial_monmap 2024-03-29T11:25:43.955 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.970+0000 7fc82f59e640 10 monclient: monmap: 2024-03-29T11:25:43.955 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:25:43.955 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:25:43.956 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:25:43.971646+0000 2024-03-29T11:25:43.956 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:25:43.971646+0000 2024-03-29T11:25:43.956 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:43.956 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:43.956 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:25:43.956 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:25:43.956 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:25:43.956 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:43.958 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.971+0000 7fc82f59e640 1 Processor -- start 2024-03-29T11:25:43.958 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.971+0000 7fc82f59e640 1 -- start start 2024-03-29T11:25:43.959 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.971+0000 7fc82f59e640 10 monclient: init 2024-03-29T11:25:43.959 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.973+0000 7fc82f59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:43.959 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.973+0000 7fc82f59e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:25:43.959 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.973+0000 7fc82f59e640 10 monclient(hunting): picked mon.noname-c con 0x7fc8280389e0 addr v1:172.21.15.94:6789/0 2024-03-29T11:25:43.959 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.973+0000 7fc82f59e640 10 monclient(hunting): picked mon.noname-a con 0x7fc828038dc0 addr v1:172.21.15.17:6789/0 2024-03-29T11:25:43.959 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.973+0000 7fc82f59e640 10 monclient(hunting): picked mon.noname-b con 0x7fc828030660 addr v1:172.21.15.17:6790/0 2024-03-29T11:25:43.959 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.973+0000 7fc82f59e640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8280c54c0 con 0x7fc828038dc0 2024-03-29T11:25:43.959 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.973+0000 7fc82f59e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc828036430 con 0x7fc8280389e0 2024-03-29T11:25:43.959 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.973+0000 7fc82f59e640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc828039840 con 0x7fc828030660 2024-03-29T11:25:43.959 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.973+0000 7fc82f59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:43.959 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.973+0000 7fc82f59e640 10 monclient(hunting): authenticate will time out at 1250.178080s 2024-03-29T11:25:43.959 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.973+0000 7fc82ed9d640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7fc828030660 0x7fc8280d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:55642/0 (socket says 172.21.15.17:55642) 2024-03-29T11:25:43.959 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.973+0000 7fc82ed9d640 1 -- 172.21.15.17:0/2495334642 learned_addr learned my addr 172.21.15.17:0/2495334642 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:25:43.960 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.976+0000 7fc82d59a640 1 -- 172.21.15.17:0/2495334642 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 858720074 0 0) 0x7fc828039840 con 0x7fc828030660 2024-03-29T11:25:43.960 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.976+0000 7fc82d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:25:43.961 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.976+0000 7fc82d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:43.961 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.976+0000 7fc82d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:43.961 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.976+0000 7fc82d59a640 10 monclient(hunting): my global_id is 4502 2024-03-29T11:25:43.961 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.976+0000 7fc82d59a640 1 -- 172.21.15.17:0/2495334642 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc818003520 con 0x7fc828030660 2024-03-29T11:25:43.967 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.979+0000 7fc82d59a640 1 -- 172.21.15.17:0/2495334642 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3863524961 0 0) 0x7fc818003520 con 0x7fc828030660 2024-03-29T11:25:43.967 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.980+0000 7fc82d59a640 1 -- 172.21.15.17:0/2495334642 >> v1:172.21.15.94:6789/0 conn(0x7fc8280389e0 legacy=0x7fc828033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:25:43.967 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.980+0000 7fc82d59a640 1 -- 172.21.15.17:0/2495334642 >> v1:172.21.15.17:6789/0 conn(0x7fc828038dc0 legacy=0x7fc8280cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:43.967 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.980+0000 7fc82d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:25:43.968 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.980+0000 7fc82d59a640 1 monclient: found mon.noname-b 2024-03-29T11:25:43.968 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.980+0000 7fc82d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:43.968 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.980+0000 7fc82d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.17:6790/0 2024-03-29T11:25:43.968 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.980+0000 7fc82d59a640 1 -- 172.21.15.17:0/2495334642 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc828035690 con 0x7fc828030660 2024-03-29T11:25:43.968 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.980+0000 7fc82d59a640 10 monclient: _finish_auth 0 2024-03-29T11:25:43.968 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.980+0000 7fc82d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:25:43.968 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.980+0000 7fc82d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:43.968 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.980+0000 7fc82d59a640 1 -- 172.21.15.17:0/2495334642 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fc824001690 con 0x7fc828030660 2024-03-29T11:25:43.968 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.980+0000 7fc82f59e640 5 monclient: authenticate success, global_id 4502 2024-03-29T11:25:43.968 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.980+0000 7fc82f59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:43.968 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.981+0000 7fc82d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:43.968 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.981+0000 7fc82d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:43.968 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.981+0000 7fc82d59a640 10 monclient: dump: 2024-03-29T11:25:43.968 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:25:43.969 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:25:43.969 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:43.969 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:43.969 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:43.969 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:43.969 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:25:43.969 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:25:43.969 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:25:43.969 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:43.969 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.981+0000 7fc82f59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:43.969 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.981+0000 7fc82d59a640 1 -- 172.21.15.17:0/2495334642 <== mon.2 v1:172.21.15.17:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc824001860 con 0x7fc828030660 2024-03-29T11:25:43.969 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.981+0000 7fc82d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:43.969 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.981+0000 7fc82d59a640 1 -- 172.21.15.17:0/2495334642 <== mon.2 v1:172.21.15.17:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fc824001cb0 con 0x7fc828030660 2024-03-29T11:25:43.969 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.981+0000 7fc82d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:43.970 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.981+0000 7fc82d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:43.970 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.981+0000 7fc82d59a640 10 monclient: dump: 2024-03-29T11:25:43.970 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:25:43.970 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:25:43.970 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:43.970 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:43.970 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:43.970 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:43.970 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:25:43.970 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:25:43.970 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:25:43.970 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:43.970 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.981+0000 7fc82f59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:43.971 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.981+0000 7fc82f59e640 4 set_mon_vals no callback set 2024-03-29T11:25:43.971 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.982+0000 7fc82f59e640 10 monclient: shutdown 2024-03-29T11:25:43.971 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.982+0000 7fc82f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:43.971 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.982+0000 7fc82f59e640 1 -- 172.21.15.17:0/2495334642 >> v1:172.21.15.17:6790/0 conn(0x7fc828030660 legacy=0x7fc8280d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:43.971 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.982+0000 7fc82f59e640 1 -- 172.21.15.17:0/2495334642 shutdown_connections 2024-03-29T11:25:43.971 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.982+0000 7fc82f59e640 1 -- 172.21.15.17:0/2495334642 >> 172.21.15.17:0/2495334642 conn(0x7fc828005c00 msgr2=0x7fc828006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:43.972 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.983+0000 7fc82f59e640 1 -- 172.21.15.17:0/2495334642 shutdown_connections 2024-03-29T11:25:43.972 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.985+0000 7fc82f59e640 1 -- 172.21.15.17:0/2495334642 wait complete. 2024-03-29T11:25:43.972 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.987+0000 7fc82f59e640 10 monclient: build_initial_monmap 2024-03-29T11:25:43.972 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.987+0000 7fc82f59e640 10 monclient: monmap: 2024-03-29T11:25:43.972 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:25:43.972 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:43.973 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:25:43.973 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:25:43.973 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:43.973 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:43.973 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:25:43.973 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:25:43.973 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:25:43.973 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:43.973 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.987+0000 7fc82f59e640 1 librados: starting msgr at 2024-03-29T11:25:43.973 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.987+0000 7fc82f59e640 1 librados: starting objecter 2024-03-29T11:25:43.973 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.988+0000 7fc82f59e640 1 Processor -- start 2024-03-29T11:25:43.973 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.988+0000 7fc82f59e640 1 -- start start 2024-03-29T11:25:43.973 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.988+0000 7fc82f59e640 1 librados: setting wanted keys 2024-03-29T11:25:43.973 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.988+0000 7fc82f59e640 1 librados: calling monclient init 2024-03-29T11:25:43.973 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.988+0000 7fc82f59e640 10 monclient: init 2024-03-29T11:25:43.974 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.989+0000 7fc82f59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:43.974 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.989+0000 7fc82f59e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:25:43.974 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.989+0000 7fc82f59e640 10 monclient(hunting): picked mon.noname-c con 0x7fc828030660 addr v1:172.21.15.17:6790/0 2024-03-29T11:25:43.977 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.992+0000 7fc82dd9b640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fc8280389e0 0x7fc828159ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:59811/0 (socket says 172.21.15.17:59811) 2024-03-29T11:25:43.977 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.992+0000 7fc82e59c640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7fc828030660 0x7fc82815cbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:35258/0 (socket says 172.21.15.17:35258) 2024-03-29T11:25:43.977 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.992+0000 7fc82dd9b640 1 -- 172.21.15.17:0/4050201254 learned_addr learned my addr 172.21.15.17:0/4050201254 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:25:43.977 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.993+0000 7fc81effd640 10 client.?.objecter ms_handle_connect 0x7fc8280389e0 2024-03-29T11:25:43.977 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.993+0000 7fc81effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:43.977 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.993+0000 7fc82f59e640 10 monclient(hunting): picked mon.noname-a con 0x7fc8280389e0 addr v1:172.21.15.17:6789/0 2024-03-29T11:25:43.978 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.993+0000 7fc81effd640 10 client.?.objecter ms_handle_connect 0x7fc828030660 2024-03-29T11:25:43.978 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.993+0000 7fc81effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:43.978 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.994+0000 7fc82f59e640 10 monclient(hunting): picked mon.noname-b con 0x7fc8280c8410 addr v1:172.21.15.94:6789/0 2024-03-29T11:25:43.978 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.994+0000 7fc82f59e640 1 -- 172.21.15.17:0/4050201254 --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc82802d680 con 0x7fc8280389e0 2024-03-29T11:25:43.978 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.994+0000 7fc82f59e640 1 -- 172.21.15.17:0/4050201254 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc82802d1c0 con 0x7fc8280c8410 2024-03-29T11:25:43.978 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.994+0000 7fc82f59e640 1 -- 172.21.15.17:0/4050201254 --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc82815a600 con 0x7fc828030660 2024-03-29T11:25:43.979 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.994+0000 7fc82f59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:43.979 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.994+0000 7fc82f59e640 10 monclient(hunting): authenticate will time out at 1250.199302s 2024-03-29T11:25:43.980 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.996+0000 7fc81effd640 1 -- 172.21.15.17:0/4050201254 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1819034646 0 0) 0x7fc82802d680 con 0x7fc8280389e0 2024-03-29T11:25:43.981 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.996+0000 7fc81effd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:43.981 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.996+0000 7fc81effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:43.981 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.996+0000 7fc81effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:43.981 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.996+0000 7fc81effd640 10 monclient(hunting): my global_id is 4461 2024-03-29T11:25:43.981 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.996+0000 7fc81effd640 1 -- 172.21.15.17:0/4050201254 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc810004eb0 con 0x7fc8280389e0 2024-03-29T11:25:43.981 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.996+0000 7fc81effd640 10 client.?.objecter ms_handle_connect 0x7fc8280c8410 2024-03-29T11:25:43.981 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.996+0000 7fc81effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:43.981 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.996+0000 7fc81effd640 1 -- 172.21.15.17:0/4050201254 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2928086677 0 0) 0x7fc82802d1c0 con 0x7fc8280c8410 2024-03-29T11:25:43.981 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.996+0000 7fc81effd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:43.981 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.996+0000 7fc81effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:43.981 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.996+0000 7fc81effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:43.981 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.996+0000 7fc81effd640 10 monclient(hunting): my global_id is 4492 2024-03-29T11:25:43.982 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.996+0000 7fc81effd640 1 -- 172.21.15.17:0/4050201254 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc82802d680 con 0x7fc8280c8410 2024-03-29T11:25:43.986 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.996+0000 7fc81effd640 1 -- 172.21.15.17:0/4050201254 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 54170021 0 0) 0x7fc810004eb0 con 0x7fc8280389e0 2024-03-29T11:25:43.986 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.997+0000 7fc81effd640 1 -- 172.21.15.17:0/4050201254 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc82802d1c0 con 0x7fc8280389e0 2024-03-29T11:25:43.986 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.997+0000 7fc81effd640 1 -- 172.21.15.17:0/4050201254 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fc83004fdd0 con 0x7fc8280389e0 2024-03-29T11:25:43.986 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.997+0000 7fc81effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:43.986 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.997+0000 7fc81effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:43.986 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.997+0000 7fc81effd640 10 monclient(hunting): dump: 2024-03-29T11:25:43.986 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:25:43.987 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:25:43.987 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:43.987 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:43.987 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:43.987 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:43.987 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:25:43.987 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:25:43.987 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:25:43.987 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:43.988 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.997+0000 7fc81effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:43.988 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.997+0000 7fc81effd640 1 -- 172.21.15.17:0/4050201254 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 510718134 0 0) 0x7fc82802d680 con 0x7fc8280c8410 2024-03-29T11:25:43.988 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.997+0000 7fc81effd640 1 -- 172.21.15.17:0/4050201254 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc810004eb0 con 0x7fc8280c8410 2024-03-29T11:25:43.988 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.997+0000 7fc81effd640 1 -- 172.21.15.17:0/4050201254 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fc8240019d0 con 0x7fc8280c8410 2024-03-29T11:25:43.988 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.997+0000 7fc81effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:43.988 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.997+0000 7fc81effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:43.988 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.997+0000 7fc81effd640 10 monclient(hunting): dump: 2024-03-29T11:25:43.988 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:25:43.988 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:25:43.988 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:43.988 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:43.989 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:43.989 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:43.989 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:25:43.989 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:25:43.989 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:25:43.989 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:43.989 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.997+0000 7fc81effd640 1 -- 172.21.15.17:0/4050201254 <== mon.1 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1830441732 0 0) 0x7fc810004eb0 con 0x7fc8280c8410 2024-03-29T11:25:43.989 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.997+0000 7fc81effd640 1 -- 172.21.15.17:0/4050201254 >> v1:172.21.15.17:6790/0 conn(0x7fc828030660 legacy=0x7fc82815cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:43.989 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.997+0000 7fc81effd640 1 -- 172.21.15.17:0/4050201254 >> v1:172.21.15.17:6789/0 conn(0x7fc8280389e0 legacy=0x7fc828159ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:43.989 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.997+0000 7fc81effd640 10 monclient: _finish_hunting 0 2024-03-29T11:25:43.989 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.997+0000 7fc81effd640 1 monclient: found mon.b 2024-03-29T11:25:43.989 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.998+0000 7fc81effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:43.989 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.998+0000 7fc81effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:25:43.989 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.998+0000 7fc81effd640 1 -- 172.21.15.17:0/4050201254 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc82815a790 con 0x7fc8280c8410 2024-03-29T11:25:43.989 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.998+0000 7fc81effd640 10 monclient: _finish_auth 0 2024-03-29T11:25:43.990 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.998+0000 7fc81effd640 10 monclient: _check_auth_tickets 2024-03-29T11:25:43.990 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.998+0000 7fc81effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:43.990 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.998+0000 7fc82f59e640 5 monclient: authenticate success, global_id 4492 2024-03-29T11:25:43.990 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.998+0000 7fc82f59e640 10 monclient: _renew_subs 2024-03-29T11:25:43.990 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.998+0000 7fc82f59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:25:43.990 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.998+0000 7fc82f59e640 1 -- 172.21.15.17:0/4050201254 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8280c7dc0 con 0x7fc8280c8410 2024-03-29T11:25:43.990 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.998+0000 7fc81effd640 1 -- 172.21.15.17:0/4050201254 <== mon.1 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc824001c70 con 0x7fc8280c8410 2024-03-29T11:25:43.990 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.998+0000 7fc81effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:43.990 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.998+0000 7fc81effd640 1 -- 172.21.15.17:0/4050201254 <== mon.1 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fc824004d90 con 0x7fc8280c8410 2024-03-29T11:25:43.990 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.998+0000 7fc81effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:43.990 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.998+0000 7fc81effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:43.990 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.998+0000 7fc81effd640 10 monclient: dump: 2024-03-29T11:25:43.990 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:25:43.990 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:25:43.991 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:43.991 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:43.991 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:43.991 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:43.991 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:25:43.991 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:25:43.991 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:25:43.991 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:43.991 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.998+0000 7fc82f59e640 10 client.4492.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:43.991 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.998+0000 7fc82f59e640 10 monclient: _renew_subs 2024-03-29T11:25:43.991 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.998+0000 7fc82f59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:25:43.991 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.998+0000 7fc82f59e640 1 -- 172.21.15.17:0/4050201254 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc82802f760 con 0x7fc8280c8410 2024-03-29T11:25:43.991 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.998+0000 7fc82f59e640 1 librados: init done 2024-03-29T11:25:43.991 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:43.999+0000 7fc82f59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:43.991 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:44.000+0000 7fc82f59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:43.991 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:44.000+0000 7fc82f59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:25:43.992 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:44.000+0000 7fc82f59e640 1 -- 172.21.15.17:0/4050201254 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc828007b90 con 0x7fc8280c8410 2024-03-29T11:25:43.992 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:44.000+0000 7fc82cd99640 4 set_mon_vals no callback set 2024-03-29T11:25:43.992 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:44.000+0000 7fc81effd640 1 -- 172.21.15.17:0/4050201254 <== mon.1 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7fc82401ad60 con 0x7fc8280c8410 2024-03-29T11:25:43.992 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:44.001+0000 7fc81effd640 1 -- 172.21.15.17:0/4050201254 <== mon.1 v1:172.21.15.94:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7fc824084910 con 0x7fc8280c8410 2024-03-29T11:25:43.992 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:44.001+0000 7fc81effd640 10 client.4492.objecter ms_dispatch 0x7fc830000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:43.992 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:44.001+0000 7fc81effd640 3 client.4492.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:43.992 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:44.001+0000 7fc81effd640 3 client.4492.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:43.992 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:44.001+0000 7fc81effd640 20 client.4492.objecter dump_active .. 0 homeless 2024-03-29T11:25:43.993 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:44.002+0000 7fc82e59c640 10 monclient: get_auth_request con 0x7fc81006dc80 auth_method 0 2024-03-29T11:25:43.993 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:44.002+0000 7fc81effd640 10 client.4492.objecter ms_handle_connect 0x7fc81006dc80 2024-03-29T11:25:43.994 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:44.007+0000 7fc81effd640 1 -- 172.21.15.17:0/4050201254 <== mon.1 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc8240501f0 con 0x7fc8280c8410 2024-03-29T11:25:43.994 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:44.007+0000 7fc81effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:43.994 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:44.007+0000 7fc81effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:44.326 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:44.341+0000 7fc82f59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:44.326 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:44.341+0000 7fc82f59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:44.326 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:44.341+0000 7fc82f59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:25:44.326 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:44.341+0000 7fc82f59e640 1 -- 172.21.15.17:0/4050201254 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc8280c8000 con 0x7fc8280c8410 2024-03-29T11:25:44.328 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:44.344+0000 7fc81effd640 1 -- 172.21.15.17:0/4050201254 <== mon.1 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7fc824052120 con 0x7fc8280c8410 2024-03-29T11:25:44.328 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:44.344+0000 7fc81effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:44.329 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:44.344+0000 7fc81effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:44.329 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:25:44.329 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:44.331 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:44.347+0000 7fc82f59e640 10 librados: watch_flush enter 2024-03-29T11:25:44.332 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:44.347+0000 7fc82f59e640 10 librados: watch_flush exit 2024-03-29T11:25:44.332 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:44.347+0000 7fc82f59e640 20 client.4492.objecter shutdown clearing up homeless session... 2024-03-29T11:25:44.332 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:44.347+0000 7fc82f59e640 10 client.4492.objecter successfully canceled tick 2024-03-29T11:25:44.332 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:44.347+0000 7fc82f59e640 1 -- 172.21.15.17:0/4050201254 >> v1:172.21.15.17:6816/3236185052 conn(0x7fc81006dc80 legacy=0x7fc810070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:44.332 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:44.347+0000 7fc82f59e640 10 monclient: shutdown 2024-03-29T11:25:44.332 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:44.347+0000 7fc82f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:44.332 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:44.347+0000 7fc82f59e640 1 -- 172.21.15.17:0/4050201254 >> v1:172.21.15.94:6789/0 conn(0x7fc8280c8410 legacy=0x7fc8281684d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:44.332 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:44.347+0000 7fc82f59e640 1 -- 172.21.15.17:0/4050201254 shutdown_connections 2024-03-29T11:25:44.333 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:44.348+0000 7fc82f59e640 1 -- 172.21.15.17:0/4050201254 >> 172.21.15.17:0/4050201254 conn(0x7fc828005c00 msgr2=0x7fc828034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:44.333 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:44.348+0000 7fc82f59e640 1 -- 172.21.15.17:0/4050201254 shutdown_connections 2024-03-29T11:25:44.333 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:44.348+0000 7fc82f59e640 1 -- 172.21.15.17:0/4050201254 wait complete. 2024-03-29T11:25:44.333 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:44.348+0000 7fc82f59e640 1 librados: shutdown 2024-03-29T11:25:44.349 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:51.156 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:51.294 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.309+0000 7f7880ad6640 10 monclient: get_monmap_and_config 2024-03-29T11:25:51.294 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.309+0000 7f7880ad6640 10 monclient: build_initial_monmap 2024-03-29T11:25:51.294 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.309+0000 7f7880ad6640 10 monclient: monmap: 2024-03-29T11:25:51.294 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:25:51.294 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:25:51.295 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:25:51.310501+0000 2024-03-29T11:25:51.295 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:25:51.310501+0000 2024-03-29T11:25:51.295 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:51.295 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:51.295 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:25:51.295 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:25:51.295 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:25:51.295 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:51.296 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.310+0000 7f7880ad6640 1 Processor -- start 2024-03-29T11:25:51.296 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.310+0000 7f7880ad6640 1 -- start start 2024-03-29T11:25:51.297 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.310+0000 7f7880ad6640 10 monclient: init 2024-03-29T11:25:51.297 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.311+0000 7f7880ad6640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:51.297 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.311+0000 7f7880ad6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:25:51.297 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.311+0000 7f7880ad6640 10 monclient(hunting): picked mon.noname-c con 0x7f787c1422b0 addr v1:172.21.15.94:6789/0 2024-03-29T11:25:51.297 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.311+0000 7f787a59c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f787c1422b0 0x7f787c13f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:46926/0 (socket says 172.21.15.17:46926) 2024-03-29T11:25:51.297 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.311+0000 7f787a59c640 1 -- 172.21.15.17:0/2705789916 learned_addr learned my addr 172.21.15.17:0/2705789916 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:25:51.297 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.311+0000 7f7880ad6640 10 monclient(hunting): picked mon.noname-a con 0x7f787c142690 addr v1:172.21.15.17:6789/0 2024-03-29T11:25:51.297 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.312+0000 7f7880ad6640 10 monclient(hunting): picked mon.noname-b con 0x7f787c0b17d0 addr v1:172.21.15.17:6790/0 2024-03-29T11:25:51.297 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.312+0000 7f7880ad6640 1 -- 172.21.15.17:0/2705789916 --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f787c142c80 con 0x7f787c142690 2024-03-29T11:25:51.297 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.312+0000 7f7880ad6640 1 -- 172.21.15.17:0/2705789916 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f787c0aae40 con 0x7f787c1422b0 2024-03-29T11:25:51.297 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.312+0000 7f7880ad6640 1 -- 172.21.15.17:0/2705789916 --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f787c13fbd0 con 0x7f787c0b17d0 2024-03-29T11:25:51.297 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.312+0000 7f7880ad6640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:51.297 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.312+0000 7f7880ad6640 10 monclient(hunting): authenticate will time out at 1257.517095s 2024-03-29T11:25:51.299 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.313+0000 7f7879d9b640 1 -- 172.21.15.17:0/2705789916 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3625160497 0 0) 0x7f787c0aae40 con 0x7f787c1422b0 2024-03-29T11:25:51.299 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.313+0000 7f7879d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:25:51.299 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.313+0000 7f7879d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:51.299 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.313+0000 7f7879d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:51.299 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.313+0000 7f7879d9b640 10 monclient(hunting): my global_id is 4495 2024-03-29T11:25:51.299 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.313+0000 7f7879d9b640 1 -- 172.21.15.17:0/2705789916 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7868003520 con 0x7f787c1422b0 2024-03-29T11:25:51.299 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.314+0000 7f7879d9b640 1 -- 172.21.15.17:0/2705789916 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1544794648 0 0) 0x7f7868003520 con 0x7f787c1422b0 2024-03-29T11:25:51.299 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.314+0000 7f7879d9b640 1 -- 172.21.15.17:0/2705789916 >> v1:172.21.15.17:6790/0 conn(0x7f787c0b17d0 legacy=0x7f787c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:51.299 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.314+0000 7f7879d9b640 1 -- 172.21.15.17:0/2705789916 >> v1:172.21.15.17:6789/0 conn(0x7f787c142690 legacy=0x7f787c0b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:51.300 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.315+0000 7f7879d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:25:51.301 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.315+0000 7f7879d9b640 1 monclient: found mon.noname-c 2024-03-29T11:25:51.301 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.315+0000 7f7879d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:51.301 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.315+0000 7f7879d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6789/0 2024-03-29T11:25:51.301 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.315+0000 7f7879d9b640 1 -- 172.21.15.17:0/2705789916 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f787c14f920 con 0x7f787c1422b0 2024-03-29T11:25:51.301 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.315+0000 7f7879d9b640 10 monclient: _finish_auth 0 2024-03-29T11:25:51.301 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.315+0000 7f7879d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:25:51.301 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.315+0000 7f7879d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:51.301 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.315+0000 7f7879d9b640 1 -- 172.21.15.17:0/2705789916 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f786c001a10 con 0x7f787c1422b0 2024-03-29T11:25:51.301 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.315+0000 7f7879d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:51.301 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.315+0000 7f7879d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:51.301 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.315+0000 7f7879d9b640 10 monclient: dump: 2024-03-29T11:25:51.301 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:25:51.302 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:25:51.302 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:51.302 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:51.302 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:51.302 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:51.302 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:25:51.302 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:25:51.302 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:25:51.302 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:51.302 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.315+0000 7f7880ad6640 5 monclient: authenticate success, global_id 4495 2024-03-29T11:25:51.302 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.315+0000 7f7880ad6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:51.302 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.315+0000 7f7879d9b640 1 -- 172.21.15.17:0/2705789916 <== mon.1 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f786c001ca0 con 0x7f787c1422b0 2024-03-29T11:25:51.302 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.315+0000 7f7879d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:51.302 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.316+0000 7f7879d9b640 1 -- 172.21.15.17:0/2705789916 <== mon.1 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f786c0020b0 con 0x7f787c1422b0 2024-03-29T11:25:51.303 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.316+0000 7f7880ad6640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:51.303 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.316+0000 7f7880ad6640 4 set_mon_vals no callback set 2024-03-29T11:25:51.305 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.316+0000 7f7879d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:51.305 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.316+0000 7f7879d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:51.305 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.316+0000 7f7879d9b640 10 monclient: dump: 2024-03-29T11:25:51.305 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:25:51.305 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:25:51.305 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:51.305 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:51.305 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:51.305 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:51.305 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:25:51.306 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:25:51.306 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:25:51.306 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:51.306 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.316+0000 7f7880ad6640 10 monclient: shutdown 2024-03-29T11:25:51.306 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.316+0000 7f7880ad6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:51.306 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.316+0000 7f7880ad6640 1 -- 172.21.15.17:0/2705789916 >> v1:172.21.15.94:6789/0 conn(0x7f787c1422b0 legacy=0x7f787c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:51.306 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.317+0000 7f7880ad6640 1 -- 172.21.15.17:0/2705789916 shutdown_connections 2024-03-29T11:25:51.306 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.317+0000 7f7880ad6640 1 -- 172.21.15.17:0/2705789916 >> 172.21.15.17:0/2705789916 conn(0x7f787c066a00 msgr2=0x7f787c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:51.306 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.317+0000 7f7880ad6640 1 -- 172.21.15.17:0/2705789916 shutdown_connections 2024-03-29T11:25:51.306 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.317+0000 7f7880ad6640 1 -- 172.21.15.17:0/2705789916 wait complete. 2024-03-29T11:25:51.306 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.318+0000 7f7880ad6640 10 monclient: build_initial_monmap 2024-03-29T11:25:51.306 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.318+0000 7f7880ad6640 10 monclient: monmap: 2024-03-29T11:25:51.306 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:25:51.307 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:51.307 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:25:51.307 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:25:51.307 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:51.307 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:51.307 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:25:51.307 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:25:51.307 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:25:51.307 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:51.307 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.318+0000 7f7880ad6640 1 librados: starting msgr at 2024-03-29T11:25:51.307 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.318+0000 7f7880ad6640 1 librados: starting objecter 2024-03-29T11:25:51.307 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.318+0000 7f7880ad6640 1 Processor -- start 2024-03-29T11:25:51.307 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.319+0000 7f7880ad6640 1 -- start start 2024-03-29T11:25:51.307 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.319+0000 7f7880ad6640 1 librados: setting wanted keys 2024-03-29T11:25:51.308 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.319+0000 7f7880ad6640 1 librados: calling monclient init 2024-03-29T11:25:51.308 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.319+0000 7f7880ad6640 10 monclient: init 2024-03-29T11:25:51.308 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.319+0000 7f7880ad6640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:51.308 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.319+0000 7f7880ad6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:25:51.308 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.320+0000 7f7880ad6640 10 monclient(hunting): picked mon.noname-c con 0x7f787c0b17d0 addr v1:172.21.15.17:6790/0 2024-03-29T11:25:51.308 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.320+0000 7f787a59c640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f787c0b17d0 0x7f787c14d360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:49018/0 (socket says 172.21.15.17:49018) 2024-03-29T11:25:51.308 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.320+0000 7f787a59c640 1 -- 172.21.15.17:0/3779659807 learned_addr learned my addr 172.21.15.17:0/3779659807 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:25:51.308 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.320+0000 7f78737fe640 10 client.?.objecter ms_handle_connect 0x7f787c0b17d0 2024-03-29T11:25:51.308 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.320+0000 7f78737fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:51.308 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.320+0000 7f7880ad6640 10 monclient(hunting): picked mon.noname-b con 0x7f787c1422b0 addr v1:172.21.15.94:6789/0 2024-03-29T11:25:51.308 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.320+0000 7f7880ad6640 10 monclient(hunting): picked mon.noname-a con 0x7f787c142690 addr v1:172.21.15.17:6789/0 2024-03-29T11:25:51.308 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.320+0000 7f7880ad6640 1 -- 172.21.15.17:0/3779659807 --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f787c14da70 con 0x7f787c142690 2024-03-29T11:25:51.308 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.320+0000 7f7880ad6640 1 -- 172.21.15.17:0/3779659807 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f787c1316a0 con 0x7f787c1422b0 2024-03-29T11:25:51.308 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.320+0000 7f7880ad6640 1 -- 172.21.15.17:0/3779659807 --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f787c06ed10 con 0x7f787c0b17d0 2024-03-29T11:25:51.309 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.320+0000 7f7880ad6640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:51.309 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.320+0000 7f7880ad6640 10 monclient(hunting): authenticate will time out at 1257.525422s 2024-03-29T11:25:51.311 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.321+0000 7f78737fe640 10 client.?.objecter ms_handle_connect 0x7f787c1422b0 2024-03-29T11:25:51.311 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.321+0000 7f78737fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:51.311 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.321+0000 7f78737fe640 1 -- 172.21.15.17:0/3779659807 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3042038485 0 0) 0x7f787c06ed10 con 0x7f787c0b17d0 2024-03-29T11:25:51.311 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.321+0000 7f78737fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:51.311 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.321+0000 7f78737fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:51.311 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.321+0000 7f78737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:51.311 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.321+0000 7f78737fe640 10 monclient(hunting): my global_id is 4511 2024-03-29T11:25:51.311 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.321+0000 7f78737fe640 1 -- 172.21.15.17:0/3779659807 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f785c004eb0 con 0x7f787c0b17d0 2024-03-29T11:25:51.312 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.321+0000 7f78737fe640 1 -- 172.21.15.17:0/3779659807 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1050271798 0 0) 0x7f787c1316a0 con 0x7f787c1422b0 2024-03-29T11:25:51.312 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.321+0000 7f78737fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:51.312 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.321+0000 7f78737fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:51.312 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.321+0000 7f78737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:51.312 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.321+0000 7f78737fe640 10 monclient(hunting): my global_id is 4498 2024-03-29T11:25:51.312 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.321+0000 7f78737fe640 1 -- 172.21.15.17:0/3779659807 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f787c06ed10 con 0x7f787c1422b0 2024-03-29T11:25:51.312 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.321+0000 7f78737fe640 1 -- 172.21.15.17:0/3779659807 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 286184142 0 0) 0x7f785c004eb0 con 0x7f787c0b17d0 2024-03-29T11:25:51.312 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.321+0000 7f78737fe640 1 -- 172.21.15.17:0/3779659807 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f787c1316a0 con 0x7f787c0b17d0 2024-03-29T11:25:51.312 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.321+0000 7f78737fe640 1 -- 172.21.15.17:0/3779659807 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f786c0019d0 con 0x7f787c0b17d0 2024-03-29T11:25:51.312 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.321+0000 7f78737fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:51.312 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.321+0000 7f78737fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:51.312 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.321+0000 7f78737fe640 10 monclient(hunting): dump: 2024-03-29T11:25:51.312 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:25:51.312 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:25:51.312 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:51.313 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:51.313 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:51.313 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:51.313 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:25:51.313 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:25:51.313 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:25:51.313 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:51.313 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.321+0000 7f78737fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:51.313 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.322+0000 7f78737fe640 1 -- 172.21.15.17:0/3779659807 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 327935288 0 0) 0x7f787c06ed10 con 0x7f787c1422b0 2024-03-29T11:25:51.313 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.322+0000 7f78737fe640 1 -- 172.21.15.17:0/3779659807 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f785c004eb0 con 0x7f787c1422b0 2024-03-29T11:25:51.313 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.322+0000 7f78737fe640 1 -- 172.21.15.17:0/3779659807 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f7864001710 con 0x7f787c1422b0 2024-03-29T11:25:51.313 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.322+0000 7f78737fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:51.313 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.322+0000 7f78737fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:51.313 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.322+0000 7f78737fe640 10 monclient(hunting): dump: 2024-03-29T11:25:51.314 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:25:51.314 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:25:51.314 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:51.314 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:51.314 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:51.314 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:51.314 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:25:51.314 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:25:51.314 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:25:51.314 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:51.314 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.322+0000 7f78737fe640 1 -- 172.21.15.17:0/3779659807 <== mon.2 v1:172.21.15.17:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2606346148 0 0) 0x7f787c1316a0 con 0x7f787c0b17d0 2024-03-29T11:25:51.314 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.322+0000 7f78737fe640 1 -- 172.21.15.17:0/3779659807 >> v1:172.21.15.94:6789/0 conn(0x7f787c1422b0 legacy=0x7f787c130880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:51.314 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.322+0000 7f78737fe640 1 -- 172.21.15.17:0/3779659807 >> v1:172.21.15.17:6789/0 conn(0x7f787c142690 legacy=0x7f787c130f90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:25:51.314 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.323+0000 7f78737fe640 10 monclient: _finish_hunting 0 2024-03-29T11:25:51.315 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.323+0000 7f78737fe640 1 monclient: found mon.c 2024-03-29T11:25:51.315 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.323+0000 7f78737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:51.315 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.323+0000 7f78737fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:25:51.315 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.323+0000 7f78737fe640 1 -- 172.21.15.17:0/3779659807 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f787c06eff0 con 0x7f787c0b17d0 2024-03-29T11:25:51.315 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.323+0000 7f78737fe640 10 monclient: _finish_auth 0 2024-03-29T11:25:51.315 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.323+0000 7f78737fe640 10 monclient: _check_auth_tickets 2024-03-29T11:25:51.315 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.323+0000 7f78737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:51.315 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.323+0000 7f7880ad6640 5 monclient: authenticate success, global_id 4511 2024-03-29T11:25:51.315 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.323+0000 7f7880ad6640 10 monclient: _renew_subs 2024-03-29T11:25:51.315 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.323+0000 7f7880ad6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:25:51.315 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.323+0000 7f7880ad6640 1 -- 172.21.15.17:0/3779659807 --> v1:172.21.15.17:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f787c14e040 con 0x7f787c0b17d0 2024-03-29T11:25:51.315 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.323+0000 7f7880ad6640 10 client.4511.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:51.315 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.323+0000 7f7880ad6640 10 monclient: _renew_subs 2024-03-29T11:25:51.315 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.323+0000 7f7880ad6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:25:51.315 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.323+0000 7f7880ad6640 1 -- 172.21.15.17:0/3779659807 --> v1:172.21.15.17:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f787c06e070 con 0x7f787c0b17d0 2024-03-29T11:25:51.316 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.323+0000 7f7880ad6640 1 librados: init done 2024-03-29T11:25:51.316 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.323+0000 7f78737fe640 1 -- 172.21.15.17:0/3779659807 <== mon.2 v1:172.21.15.17:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f786c001b30 con 0x7f787c0b17d0 2024-03-29T11:25:51.316 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.323+0000 7f78737fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:51.316 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.323+0000 7f787959a640 4 set_mon_vals no callback set 2024-03-29T11:25:51.316 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.324+0000 7f78737fe640 1 -- 172.21.15.17:0/3779659807 <== mon.2 v1:172.21.15.17:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f786c005150 con 0x7f787c0b17d0 2024-03-29T11:25:51.316 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.324+0000 7f78737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:51.316 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.324+0000 7f78737fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:51.316 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.324+0000 7f78737fe640 10 monclient: dump: 2024-03-29T11:25:51.316 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:25:51.316 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:25:51.316 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:51.316 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:51.316 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:51.316 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:51.317 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:25:51.317 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:25:51.317 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:25:51.317 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:51.317 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.324+0000 7f7880ad6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:51.317 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.324+0000 7f7880ad6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:51.317 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.324+0000 7f7880ad6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:25:51.317 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.324+0000 7f7880ad6640 1 -- 172.21.15.17:0/3779659807 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f787c075f10 con 0x7f787c0b17d0 2024-03-29T11:25:51.317 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.325+0000 7f78737fe640 1 -- 172.21.15.17:0/3779659807 <== mon.2 v1:172.21.15.17:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f786c01b1a0 con 0x7f787c0b17d0 2024-03-29T11:25:51.317 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.326+0000 7f78737fe640 1 -- 172.21.15.17:0/3779659807 <== mon.2 v1:172.21.15.17:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f786c084fe0 con 0x7f787c0b17d0 2024-03-29T11:25:51.317 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.326+0000 7f78737fe640 10 client.4511.objecter ms_dispatch 0x7f787c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:51.317 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.326+0000 7f78737fe640 3 client.4511.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:51.317 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.326+0000 7f78737fe640 3 client.4511.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:51.317 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.326+0000 7f78737fe640 20 client.4511.objecter dump_active .. 0 homeless 2024-03-29T11:25:51.318 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.326+0000 7f7871d9b640 10 monclient: get_auth_request con 0x7f785c06dee0 auth_method 0 2024-03-29T11:25:51.318 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.326+0000 7f78737fe640 10 client.4511.objecter ms_handle_connect 0x7f785c06dee0 2024-03-29T11:25:51.319 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.333+0000 7f78737fe640 1 -- 172.21.15.17:0/3779659807 <== mon.2 v1:172.21.15.17:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f786c0508c0 con 0x7f787c0b17d0 2024-03-29T11:25:51.319 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.333+0000 7f78737fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:51.319 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.333+0000 7f78737fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:51.637 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.652+0000 7f7880ad6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:51.638 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.652+0000 7f7880ad6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:51.638 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.652+0000 7f7880ad6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:25:51.638 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.652+0000 7f7880ad6640 1 -- 172.21.15.17:0/3779659807 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f787c0aab60 con 0x7f787c0b17d0 2024-03-29T11:25:51.639 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.655+0000 7f78737fe640 1 -- 172.21.15.17:0/3779659807 <== mon.2 v1:172.21.15.17:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f786c0527f0 con 0x7f787c0b17d0 2024-03-29T11:25:51.639 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.655+0000 7f78737fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:51.640 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.655+0000 7f78737fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:51.640 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:25:51.640 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:51.644 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.659+0000 7f7880ad6640 10 librados: watch_flush enter 2024-03-29T11:25:51.644 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.659+0000 7f7880ad6640 10 librados: watch_flush exit 2024-03-29T11:25:51.644 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.659+0000 7f7880ad6640 20 client.4511.objecter shutdown clearing up homeless session... 2024-03-29T11:25:51.644 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.659+0000 7f7880ad6640 10 client.4511.objecter successfully canceled tick 2024-03-29T11:25:51.644 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.659+0000 7f7880ad6640 1 -- 172.21.15.17:0/3779659807 >> v1:172.21.15.17:6816/3236185052 conn(0x7f785c06dee0 legacy=0x7f785c0703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:51.644 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.659+0000 7f7880ad6640 10 monclient: shutdown 2024-03-29T11:25:51.645 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.659+0000 7f7880ad6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:51.645 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.659+0000 7f7880ad6640 1 -- 172.21.15.17:0/3779659807 >> v1:172.21.15.17:6790/0 conn(0x7f787c0b17d0 legacy=0x7f787c14d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:51.645 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.659+0000 7f7880ad6640 1 -- 172.21.15.17:0/3779659807 shutdown_connections 2024-03-29T11:25:51.645 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.659+0000 7f7880ad6640 1 -- 172.21.15.17:0/3779659807 >> 172.21.15.17:0/3779659807 conn(0x7f787c066a00 msgr2=0x7f787c13d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:51.645 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.659+0000 7f7880ad6640 1 -- 172.21.15.17:0/3779659807 shutdown_connections 2024-03-29T11:25:51.645 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.659+0000 7f7880ad6640 1 -- 172.21.15.17:0/3779659807 wait complete. 2024-03-29T11:25:51.645 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:51.659+0000 7f7880ad6640 1 librados: shutdown 2024-03-29T11:25:51.660 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:58.470 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:58.637 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.651+0000 7f6d64f03640 10 monclient: get_monmap_and_config 2024-03-29T11:25:58.637 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.651+0000 7f6d64f03640 10 monclient: build_initial_monmap 2024-03-29T11:25:58.637 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.651+0000 7f6d64f03640 10 monclient: monmap: 2024-03-29T11:25:58.637 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:25:58.637 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:25:58.637 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:25:58.652420+0000 2024-03-29T11:25:58.637 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:25:58.652420+0000 2024-03-29T11:25:58.637 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:58.637 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:58.637 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:25:58.637 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:25:58.638 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:25:58.638 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:58.640 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.655+0000 7f6d64f03640 1 Processor -- start 2024-03-29T11:25:58.640 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.655+0000 7f6d64f03640 1 -- start start 2024-03-29T11:25:58.640 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.655+0000 7f6d64f03640 10 monclient: init 2024-03-29T11:25:58.644 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.656+0000 7f6d64f03640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:58.644 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.656+0000 7f6d64f03640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:25:58.644 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.656+0000 7f6d64f03640 10 monclient(hunting): picked mon.noname-c con 0x7f6d5802b4c0 addr v1:172.21.15.94:6789/0 2024-03-29T11:25:58.644 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.656+0000 7f6d64f03640 10 monclient(hunting): picked mon.noname-a con 0x7f6d5800b8a0 addr v1:172.21.15.17:6789/0 2024-03-29T11:25:58.644 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.659+0000 7f6d5f7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f6d5802b4c0 0x7f6d58006890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:55413/0 (socket says 172.21.15.17:55413) 2024-03-29T11:25:58.645 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.659+0000 7f6d64f03640 10 monclient(hunting): picked mon.noname-b con 0x7f6d5802a5a0 addr v1:172.21.15.17:6790/0 2024-03-29T11:25:58.645 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.659+0000 7f6d64f03640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d5802c5c0 con 0x7f6d5800b8a0 2024-03-29T11:25:58.645 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.659+0000 7f6d64f03640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d580c11a0 con 0x7f6d5802b4c0 2024-03-29T11:25:58.645 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.659+0000 7f6d64f03640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d5802afa0 con 0x7f6d5802a5a0 2024-03-29T11:25:58.645 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.659+0000 7f6d64f03640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:58.645 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.659+0000 7f6d64f03640 10 monclient(hunting): authenticate will time out at 1264.864339s 2024-03-29T11:25:58.645 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.659+0000 7f6d5f7fe640 1 -- 172.21.15.17:0/1575197619 learned_addr learned my addr 172.21.15.17:0/1575197619 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:25:58.647 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.662+0000 7f6d5e7fc640 1 -- 172.21.15.17:0/1575197619 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2811575207 0 0) 0x7f6d5802afa0 con 0x7f6d5802a5a0 2024-03-29T11:25:58.648 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.662+0000 7f6d5e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:58.648 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.662+0000 7f6d5e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:58.648 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.662+0000 7f6d5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:58.648 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.662+0000 7f6d5e7fc640 10 monclient(hunting): my global_id is 4514 2024-03-29T11:25:58.648 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.662+0000 7f6d5e7fc640 1 -- 172.21.15.17:0/1575197619 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d50003520 con 0x7f6d5802a5a0 2024-03-29T11:25:58.648 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.663+0000 7f6d5e7fc640 1 -- 172.21.15.17:0/1575197619 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 417673556 0 0) 0x7f6d5802c5c0 con 0x7f6d5800b8a0 2024-03-29T11:25:58.648 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.663+0000 7f6d5e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:58.648 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.663+0000 7f6d5e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:58.648 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.663+0000 7f6d5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:58.648 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.663+0000 7f6d5e7fc640 10 monclient(hunting): my global_id is 4464 2024-03-29T11:25:58.649 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.663+0000 7f6d5e7fc640 1 -- 172.21.15.17:0/1575197619 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d5802afa0 con 0x7f6d5800b8a0 2024-03-29T11:25:58.649 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.664+0000 7f6d5e7fc640 1 -- 172.21.15.17:0/1575197619 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1470979834 0 0) 0x7f6d5802afa0 con 0x7f6d5800b8a0 2024-03-29T11:25:58.649 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.664+0000 7f6d5e7fc640 1 -- 172.21.15.17:0/1575197619 >> v1:172.21.15.17:6790/0 conn(0x7f6d5802a5a0 legacy=0x7f6d580c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:58.649 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.664+0000 7f6d5e7fc640 1 -- 172.21.15.17:0/1575197619 >> v1:172.21.15.94:6789/0 conn(0x7f6d5802b4c0 legacy=0x7f6d58006890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:58.649 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.664+0000 7f6d5e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:25:58.649 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.664+0000 7f6d5e7fc640 1 monclient: found mon.noname-a 2024-03-29T11:25:58.650 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.664+0000 7f6d5e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:58.650 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.664+0000 7f6d5e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.17:6789/0 2024-03-29T11:25:58.650 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.664+0000 7f6d5e7fc640 1 -- 172.21.15.17:0/1575197619 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d580329c0 con 0x7f6d5800b8a0 2024-03-29T11:25:58.650 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.664+0000 7f6d5e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:25:58.650 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.664+0000 7f6d5e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:25:58.650 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.664+0000 7f6d5e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:58.650 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.664+0000 7f6d5e7fc640 1 -- 172.21.15.17:0/1575197619 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f6d6004f730 con 0x7f6d5800b8a0 2024-03-29T11:25:58.650 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.664+0000 7f6d5e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:58.650 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.664+0000 7f6d5e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:58.650 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.664+0000 7f6d5e7fc640 10 monclient: dump: 2024-03-29T11:25:58.650 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:25:58.650 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:25:58.650 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:58.651 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:58.651 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:58.651 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:58.651 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:25:58.651 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:25:58.651 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:25:58.651 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:58.651 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.664+0000 7f6d64f03640 5 monclient: authenticate success, global_id 4464 2024-03-29T11:25:58.651 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.664+0000 7f6d64f03640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:58.652 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.664+0000 7f6d5e7fc640 1 -- 172.21.15.17:0/1575197619 <== mon.0 v1:172.21.15.17:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d600008d0 con 0x7f6d5800b8a0 2024-03-29T11:25:58.652 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.664+0000 7f6d5e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:58.652 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.664+0000 7f6d5e7fc640 1 -- 172.21.15.17:0/1575197619 <== mon.0 v1:172.21.15.17:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f6d600758b0 con 0x7f6d5800b8a0 2024-03-29T11:25:58.652 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.664+0000 7f6d5e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:58.652 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.664+0000 7f6d5e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:58.652 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.664+0000 7f6d5e7fc640 10 monclient: dump: 2024-03-29T11:25:58.652 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:25:58.652 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:25:58.652 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:58.652 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:58.653 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:58.653 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:58.653 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:25:58.653 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:25:58.653 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:25:58.653 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:58.653 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.664+0000 7f6d64f03640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:58.653 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.664+0000 7f6d64f03640 4 set_mon_vals no callback set 2024-03-29T11:25:58.653 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.665+0000 7f6d64f03640 10 monclient: shutdown 2024-03-29T11:25:58.653 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.665+0000 7f6d64f03640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:58.653 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.665+0000 7f6d64f03640 1 -- 172.21.15.17:0/1575197619 >> v1:172.21.15.17:6789/0 conn(0x7f6d5800b8a0 legacy=0x7f6d580c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:58.653 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.666+0000 7f6d64f03640 1 -- 172.21.15.17:0/1575197619 shutdown_connections 2024-03-29T11:25:58.653 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.666+0000 7f6d64f03640 1 -- 172.21.15.17:0/1575197619 >> 172.21.15.17:0/1575197619 conn(0x7f6d580058f0 msgr2=0x7f6d58005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:58.654 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.666+0000 7f6d64f03640 1 -- 172.21.15.17:0/1575197619 shutdown_connections 2024-03-29T11:25:58.655 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.667+0000 7f6d64f03640 1 -- 172.21.15.17:0/1575197619 wait complete. 2024-03-29T11:25:58.655 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.667+0000 7f6d64f03640 10 monclient: build_initial_monmap 2024-03-29T11:25:58.655 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.667+0000 7f6d64f03640 10 monclient: monmap: 2024-03-29T11:25:58.655 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:25:58.655 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:58.655 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:25:58.655 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:25:58.655 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:58.655 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:58.655 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:25:58.655 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:25:58.655 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:25:58.655 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:58.655 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.668+0000 7f6d64f03640 1 librados: starting msgr at 2024-03-29T11:25:58.655 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.668+0000 7f6d64f03640 1 librados: starting objecter 2024-03-29T11:25:58.656 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.668+0000 7f6d64f03640 1 Processor -- start 2024-03-29T11:25:58.656 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.668+0000 7f6d64f03640 1 -- start start 2024-03-29T11:25:58.656 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.668+0000 7f6d64f03640 1 librados: setting wanted keys 2024-03-29T11:25:58.656 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.669+0000 7f6d64f03640 1 librados: calling monclient init 2024-03-29T11:25:58.656 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.669+0000 7f6d64f03640 10 monclient: init 2024-03-29T11:25:58.656 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.670+0000 7f6d64f03640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:58.656 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.670+0000 7f6d64f03640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:25:58.656 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.670+0000 7f6d64f03640 10 monclient(hunting): picked mon.noname-a con 0x7f6d5800b8a0 addr v1:172.21.15.17:6789/0 2024-03-29T11:25:58.659 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.671+0000 7f6d64f03640 10 monclient(hunting): picked mon.noname-b con 0x7f6d5802a5a0 addr v1:172.21.15.94:6789/0 2024-03-29T11:25:58.659 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.671+0000 7f6d64f03640 10 monclient(hunting): picked mon.noname-c con 0x7f6d5802b4c0 addr v1:172.21.15.17:6790/0 2024-03-29T11:25:58.659 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.671+0000 7f6d64f03640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d5802e2e0 con 0x7f6d5800b8a0 2024-03-29T11:25:58.659 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.671+0000 7f6d64f03640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d5802f430 con 0x7f6d5802a5a0 2024-03-29T11:25:58.659 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.671+0000 7f6d64f03640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d5802f6a0 con 0x7f6d5802b4c0 2024-03-29T11:25:58.659 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.671+0000 7f6d64f03640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:58.659 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.671+0000 7f6d64f03640 10 monclient(hunting): authenticate will time out at 1264.876193s 2024-03-29T11:25:58.661 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.676+0000 7f6d5f7fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f6d5800b8a0 0x7f6d580c3af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:54857/0 (socket says 172.21.15.17:54857) 2024-03-29T11:25:58.661 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.676+0000 7f6d5f7fe640 1 -- 172.21.15.17:0/1963086850 learned_addr learned my addr 172.21.15.17:0/1963086850 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:25:58.662 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.677+0000 7f6d4bfff640 10 client.?.objecter ms_handle_connect 0x7f6d5802a5a0 2024-03-29T11:25:58.662 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.677+0000 7f6d4bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:58.663 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.677+0000 7f6d4bfff640 1 -- 172.21.15.17:0/1963086850 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2293798846 0 0) 0x7f6d5802f430 con 0x7f6d5802a5a0 2024-03-29T11:25:58.664 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.677+0000 7f6d4bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:58.664 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.677+0000 7f6d4bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:58.664 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.677+0000 7f6d4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:58.664 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.677+0000 7f6d4bfff640 10 monclient(hunting): my global_id is 4501 2024-03-29T11:25:58.664 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.677+0000 7f6d4bfff640 1 -- 172.21.15.17:0/1963086850 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d44004eb0 con 0x7f6d5802a5a0 2024-03-29T11:25:58.664 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.677+0000 7f6d4bfff640 10 client.?.objecter ms_handle_connect 0x7f6d5800b8a0 2024-03-29T11:25:58.664 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.677+0000 7f6d4bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:58.664 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.678+0000 7f6d4bfff640 1 -- 172.21.15.17:0/1963086850 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 932451443 0 0) 0x7f6d5802e2e0 con 0x7f6d5800b8a0 2024-03-29T11:25:58.664 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.678+0000 7f6d4bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:58.664 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.678+0000 7f6d4bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:58.664 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.678+0000 7f6d4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:58.664 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.678+0000 7f6d4bfff640 10 monclient(hunting): my global_id is 4467 2024-03-29T11:25:58.664 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.678+0000 7f6d4bfff640 1 -- 172.21.15.17:0/1963086850 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d44006230 con 0x7f6d5800b8a0 2024-03-29T11:25:58.664 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.678+0000 7f6d4bfff640 10 client.?.objecter ms_handle_connect 0x7f6d5802b4c0 2024-03-29T11:25:58.665 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.678+0000 7f6d4bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:58.665 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.678+0000 7f6d4bfff640 1 -- 172.21.15.17:0/1963086850 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 717938220 0 0) 0x7f6d44006230 con 0x7f6d5800b8a0 2024-03-29T11:25:58.665 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.678+0000 7f6d4bfff640 1 -- 172.21.15.17:0/1963086850 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d440066a0 con 0x7f6d5800b8a0 2024-03-29T11:25:58.666 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.678+0000 7f6d4bfff640 1 -- 172.21.15.17:0/1963086850 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2623743245 0 0) 0x7f6d5802f6a0 con 0x7f6d5802b4c0 2024-03-29T11:25:58.666 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.678+0000 7f6d4bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:58.666 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.678+0000 7f6d4bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:58.666 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.678+0000 7f6d4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:58.666 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.678+0000 7f6d4bfff640 10 monclient(hunting): my global_id is 4517 2024-03-29T11:25:58.666 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.678+0000 7f6d4bfff640 1 -- 172.21.15.17:0/1963086850 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d44006b00 con 0x7f6d5802b4c0 2024-03-29T11:25:58.666 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.678+0000 7f6d4bfff640 1 -- 172.21.15.17:0/1963086850 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f6d54001930 con 0x7f6d5800b8a0 2024-03-29T11:25:58.666 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.678+0000 7f6d4bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:58.666 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.678+0000 7f6d4bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:58.666 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.678+0000 7f6d4bfff640 10 monclient(hunting): dump: 2024-03-29T11:25:58.666 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:25:58.667 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:25:58.667 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:58.667 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:58.667 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:58.667 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:58.667 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:25:58.667 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:25:58.667 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:25:58.667 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:58.667 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.678+0000 7f6d4bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:58.667 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.678+0000 7f6d4bfff640 1 -- 172.21.15.17:0/1963086850 <== mon.0 v1:172.21.15.17:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1514680744 0 0) 0x7f6d440066a0 con 0x7f6d5800b8a0 2024-03-29T11:25:58.667 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.678+0000 7f6d4bfff640 1 -- 172.21.15.17:0/1963086850 >> v1:172.21.15.17:6790/0 conn(0x7f6d5802b4c0 legacy=0x7f6d58174f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:58.667 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.678+0000 7f6d4bfff640 1 -- 172.21.15.17:0/1963086850 >> v1:172.21.15.94:6789/0 conn(0x7f6d5802a5a0 legacy=0x7f6d580bfe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:58.668 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.678+0000 7f6d4bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:25:58.668 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.678+0000 7f6d4bfff640 1 monclient: found mon.a 2024-03-29T11:25:58.668 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.678+0000 7f6d4bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:58.668 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.678+0000 7f6d4bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:25:58.668 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.678+0000 7f6d4bfff640 1 -- 172.21.15.17:0/1963086850 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d5802f980 con 0x7f6d5800b8a0 2024-03-29T11:25:58.668 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.678+0000 7f6d4bfff640 10 monclient: _finish_auth 0 2024-03-29T11:25:58.668 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.678+0000 7f6d4bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:25:58.668 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.678+0000 7f6d4bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:58.668 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.678+0000 7f6d64f03640 5 monclient: authenticate success, global_id 4467 2024-03-29T11:25:58.668 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.678+0000 7f6d64f03640 10 monclient: _renew_subs 2024-03-29T11:25:58.668 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.678+0000 7f6d64f03640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:25:58.668 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.678+0000 7f6d64f03640 1 -- 172.21.15.17:0/1963086850 --> v1:172.21.15.17:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d5802e770 con 0x7f6d5800b8a0 2024-03-29T11:25:58.668 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.679+0000 7f6d64f03640 10 client.4467.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:58.668 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.679+0000 7f6d64f03640 10 monclient: _renew_subs 2024-03-29T11:25:58.668 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.679+0000 7f6d64f03640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:25:58.669 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.679+0000 7f6d64f03640 1 -- 172.21.15.17:0/1963086850 --> v1:172.21.15.17:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d580c43e0 con 0x7f6d5800b8a0 2024-03-29T11:25:58.669 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.679+0000 7f6d64f03640 1 librados: init done 2024-03-29T11:25:58.669 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.679+0000 7f6d4bfff640 1 -- 172.21.15.17:0/1963086850 <== mon.0 v1:172.21.15.17:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d540047e0 con 0x7f6d5800b8a0 2024-03-29T11:25:58.669 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.679+0000 7f6d4bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:58.669 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.679+0000 7f6d4bfff640 1 -- 172.21.15.17:0/1963086850 <== mon.0 v1:172.21.15.17:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f6d54004af0 con 0x7f6d5800b8a0 2024-03-29T11:25:58.669 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.679+0000 7f6d4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:58.669 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.679+0000 7f6d4bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:58.669 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.679+0000 7f6d4bfff640 10 monclient: dump: 2024-03-29T11:25:58.669 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:25:58.669 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:25:58.669 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:58.669 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:25:58.669 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:58.669 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:25:58.669 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:25:58.670 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:25:58.670 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:25:58.670 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:25:58.670 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.680+0000 7f6d64f03640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:58.670 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.680+0000 7f6d64f03640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:58.670 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.680+0000 7f6d64f03640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:25:58.670 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.680+0000 7f6d64f03640 1 -- 172.21.15.17:0/1963086850 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d5802e960 con 0x7f6d5800b8a0 2024-03-29T11:25:58.671 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.682+0000 7f6d4bfff640 1 -- 172.21.15.17:0/1963086850 <== mon.0 v1:172.21.15.17:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f6d5401aac0 con 0x7f6d5800b8a0 2024-03-29T11:25:58.671 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.682+0000 7f6d4bfff640 1 -- 172.21.15.17:0/1963086850 <== mon.0 v1:172.21.15.17:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f6d54084a40 con 0x7f6d5800b8a0 2024-03-29T11:25:58.672 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.682+0000 7f6d4bfff640 10 client.4467.objecter ms_dispatch 0x7f6d60000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:58.672 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.682+0000 7f6d4bfff640 3 client.4467.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:58.672 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.682+0000 7f6d4bfff640 3 client.4467.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:58.672 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.682+0000 7f6d4bfff640 20 client.4467.objecter dump_active .. 0 homeless 2024-03-29T11:25:58.672 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.683+0000 7f6d5dffb640 4 set_mon_vals no callback set 2024-03-29T11:25:58.672 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.683+0000 7f6d5effd640 10 monclient: get_auth_request con 0x7f6d4406e320 auth_method 0 2024-03-29T11:25:58.672 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.683+0000 7f6d4bfff640 10 client.4467.objecter ms_handle_connect 0x7f6d4406e320 2024-03-29T11:25:58.676 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.691+0000 7f6d4bfff640 1 -- 172.21.15.17:0/1963086850 <== mon.0 v1:172.21.15.17:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6d54050320 con 0x7f6d5800b8a0 2024-03-29T11:25:58.676 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.691+0000 7f6d4bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:58.677 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:58.691+0000 7f6d4bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:59.009 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:59.024+0000 7f6d64f03640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:59.009 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:59.024+0000 7f6d64f03640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:59.009 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:59.024+0000 7f6d64f03640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:25:59.010 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:59.024+0000 7f6d64f03640 1 -- 172.21.15.17:0/1963086850 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d58004cd0 con 0x7f6d5800b8a0 2024-03-29T11:25:59.010 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:59.024+0000 7f6d4bfff640 1 -- 172.21.15.17:0/1963086850 <== mon.0 v1:172.21.15.17:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f6d54052250 con 0x7f6d5800b8a0 2024-03-29T11:25:59.010 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:59.024+0000 7f6d4bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:59.010 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:59.024+0000 7f6d4bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:59.013 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:25:59.013 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:59.013 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:59.028+0000 7f6d64f03640 10 librados: watch_flush enter 2024-03-29T11:25:59.013 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:59.028+0000 7f6d64f03640 10 librados: watch_flush exit 2024-03-29T11:25:59.013 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:59.028+0000 7f6d64f03640 20 client.4467.objecter shutdown clearing up homeless session... 2024-03-29T11:25:59.013 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:59.028+0000 7f6d64f03640 10 client.4467.objecter successfully canceled tick 2024-03-29T11:25:59.014 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:59.028+0000 7f6d64f03640 1 -- 172.21.15.17:0/1963086850 >> v1:172.21.15.17:6816/3236185052 conn(0x7f6d4406e320 legacy=0x7f6d440707e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:59.015 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:59.028+0000 7f6d64f03640 10 monclient: shutdown 2024-03-29T11:25:59.015 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:59.028+0000 7f6d64f03640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:59.015 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:59.029+0000 7f6d64f03640 1 -- 172.21.15.17:0/1963086850 >> v1:172.21.15.17:6789/0 conn(0x7f6d5800b8a0 legacy=0x7f6d580c3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:59.017 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:59.032+0000 7f6d64f03640 1 -- 172.21.15.17:0/1963086850 shutdown_connections 2024-03-29T11:25:59.017 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:59.032+0000 7f6d64f03640 1 -- 172.21.15.17:0/1963086850 >> 172.21.15.17:0/1963086850 conn(0x7f6d580058f0 msgr2=0x7f6d58033d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:59.018 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:59.032+0000 7f6d64f03640 1 -- 172.21.15.17:0/1963086850 shutdown_connections 2024-03-29T11:25:59.018 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:59.033+0000 7f6d64f03640 1 -- 172.21.15.17:0/1963086850 wait complete. 2024-03-29T11:25:59.021 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:25:59.036+0000 7f6d64f03640 1 librados: shutdown 2024-03-29T11:25:59.039 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:05.846 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:05.996 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.010+0000 7f012a4e6640 10 monclient: get_monmap_and_config 2024-03-29T11:26:05.996 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.010+0000 7f012a4e6640 10 monclient: build_initial_monmap 2024-03-29T11:26:05.997 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.010+0000 7f012a4e6640 10 monclient: monmap: 2024-03-29T11:26:05.997 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:26:05.997 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:26:05.997 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:26:06.011686+0000 2024-03-29T11:26:05.997 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:26:06.011686+0000 2024-03-29T11:26:05.997 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:05.997 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:05.997 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:26:05.997 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:26:05.997 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:26:05.997 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:05.998 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.012+0000 7f012a4e6640 1 Processor -- start 2024-03-29T11:26:05.998 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.012+0000 7f012a4e6640 1 -- start start 2024-03-29T11:26:05.998 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.012+0000 7f012a4e6640 10 monclient: init 2024-03-29T11:26:05.998 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.013+0000 7f012a4e6640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:05.998 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.013+0000 7f012a4e6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:26:05.998 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.013+0000 7f012a4e6640 10 monclient(hunting): picked mon.noname-c con 0x7f01240b0770 addr v1:172.21.15.94:6789/0 2024-03-29T11:26:05.999 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.013+0000 7f012a4e6640 10 monclient(hunting): picked mon.noname-a con 0x7f012414a700 addr v1:172.21.15.17:6789/0 2024-03-29T11:26:05.999 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.013+0000 7f012a4e6640 10 monclient(hunting): picked mon.noname-b con 0x7f01240ac5a0 addr v1:172.21.15.17:6790/0 2024-03-29T11:26:05.999 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.013+0000 7f012a4e6640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01240aba70 con 0x7f012414a700 2024-03-29T11:26:05.999 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.013+0000 7f012a4e6640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01240aecb0 con 0x7f01240b0770 2024-03-29T11:26:05.999 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.013+0000 7f012a4e6640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01240af150 con 0x7f01240ac5a0 2024-03-29T11:26:05.999 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.013+0000 7f012a4e6640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:05.999 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.013+0000 7f012a4e6640 10 monclient(hunting): authenticate will time out at 1272.218546s 2024-03-29T11:26:06.002 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.016+0000 7f0128a83640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f01240ac5a0 0x7f0124151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:55883/0 (socket says 172.21.15.17:55883) 2024-03-29T11:26:06.002 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.016+0000 7f0128a83640 1 -- 172.21.15.17:0/2379439865 learned_addr learned my addr 172.21.15.17:0/2379439865 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:26:06.004 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.018+0000 7f0122ffd640 1 -- 172.21.15.17:0/2379439865 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3969472751 0 0) 0x7f01240af150 con 0x7f01240ac5a0 2024-03-29T11:26:06.004 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.019+0000 7f0122ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:06.004 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.019+0000 7f0122ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:06.004 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.019+0000 7f0122ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:06.004 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.019+0000 7f0122ffd640 10 monclient(hunting): my global_id is 4520 2024-03-29T11:26:06.004 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.019+0000 7f0122ffd640 1 -- 172.21.15.17:0/2379439865 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f010c003520 con 0x7f01240ac5a0 2024-03-29T11:26:06.005 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.019+0000 7f0122ffd640 1 -- 172.21.15.17:0/2379439865 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1373683569 0 0) 0x7f01240aecb0 con 0x7f01240b0770 2024-03-29T11:26:06.005 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.019+0000 7f0122ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:06.005 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.019+0000 7f0122ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:06.005 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.019+0000 7f0122ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:06.005 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.019+0000 7f0122ffd640 10 monclient(hunting): my global_id is 4504 2024-03-29T11:26:06.005 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.019+0000 7f0122ffd640 1 -- 172.21.15.17:0/2379439865 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01240af150 con 0x7f01240b0770 2024-03-29T11:26:06.005 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.019+0000 7f0122ffd640 1 -- 172.21.15.17:0/2379439865 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 976954948 0 0) 0x7f01240aba70 con 0x7f012414a700 2024-03-29T11:26:06.005 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.019+0000 7f0122ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:06.005 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.019+0000 7f0122ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:06.005 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.019+0000 7f0122ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:06.006 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.019+0000 7f0122ffd640 10 monclient(hunting): my global_id is 4470 2024-03-29T11:26:06.006 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.019+0000 7f0122ffd640 1 -- 172.21.15.17:0/2379439865 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01240aecb0 con 0x7f012414a700 2024-03-29T11:26:06.009 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.019+0000 7f0122ffd640 1 -- 172.21.15.17:0/2379439865 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2437029082 0 0) 0x7f01240aecb0 con 0x7f012414a700 2024-03-29T11:26:06.009 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.019+0000 7f0122ffd640 1 -- 172.21.15.17:0/2379439865 >> v1:172.21.15.17:6790/0 conn(0x7f01240ac5a0 legacy=0x7f0124151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:06.009 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.019+0000 7f0122ffd640 1 -- 172.21.15.17:0/2379439865 >> v1:172.21.15.94:6789/0 conn(0x7f01240b0770 legacy=0x7f0124149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:06.009 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.019+0000 7f0122ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:26:06.009 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.019+0000 7f0122ffd640 1 monclient: found mon.noname-a 2024-03-29T11:26:06.009 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.019+0000 7f0122ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:06.009 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.019+0000 7f0122ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.17:6789/0 2024-03-29T11:26:06.009 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.019+0000 7f0122ffd640 1 -- 172.21.15.17:0/2379439865 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01240ac9d0 con 0x7f012414a700 2024-03-29T11:26:06.009 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.019+0000 7f0122ffd640 10 monclient: _finish_auth 0 2024-03-29T11:26:06.010 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.019+0000 7f0122ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:26:06.010 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.019+0000 7f0122ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:06.010 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.019+0000 7f0122ffd640 1 -- 172.21.15.17:0/2379439865 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f011c0017e0 con 0x7f012414a700 2024-03-29T11:26:06.010 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.019+0000 7f0122ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:06.010 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.019+0000 7f0122ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:06.010 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.019+0000 7f0122ffd640 10 monclient: dump: 2024-03-29T11:26:06.010 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:26:06.010 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:26:06.010 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:06.010 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:06.010 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:06.011 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:06.011 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:26:06.011 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:26:06.011 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:26:06.011 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:06.011 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.019+0000 7f012a4e6640 5 monclient: authenticate success, global_id 4470 2024-03-29T11:26:06.011 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.019+0000 7f012a4e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:06.011 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.020+0000 7f0122ffd640 1 -- 172.21.15.17:0/2379439865 <== mon.0 v1:172.21.15.17:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f011c001940 con 0x7f012414a700 2024-03-29T11:26:06.011 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.020+0000 7f0122ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:06.011 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.020+0000 7f0122ffd640 1 -- 172.21.15.17:0/2379439865 <== mon.0 v1:172.21.15.17:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f011c001d10 con 0x7f012414a700 2024-03-29T11:26:06.012 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.020+0000 7f0122ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:06.012 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.020+0000 7f0122ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:06.012 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.020+0000 7f0122ffd640 10 monclient: dump: 2024-03-29T11:26:06.012 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:26:06.012 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:26:06.012 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:06.012 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:06.012 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:06.012 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:06.012 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:26:06.012 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:26:06.012 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:26:06.012 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:06.013 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.020+0000 7f012a4e6640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:06.013 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.020+0000 7f012a4e6640 4 set_mon_vals no callback set 2024-03-29T11:26:06.013 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.021+0000 7f012a4e6640 10 monclient: shutdown 2024-03-29T11:26:06.013 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.021+0000 7f012a4e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:06.013 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.021+0000 7f012a4e6640 1 -- 172.21.15.17:0/2379439865 >> v1:172.21.15.17:6789/0 conn(0x7f012414a700 legacy=0x7f01240abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:06.013 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.021+0000 7f012a4e6640 1 -- 172.21.15.17:0/2379439865 shutdown_connections 2024-03-29T11:26:06.013 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.021+0000 7f012a4e6640 1 -- 172.21.15.17:0/2379439865 >> 172.21.15.17:0/2379439865 conn(0x7f012406b660 msgr2=0x7f012406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:06.013 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.021+0000 7f012a4e6640 1 -- 172.21.15.17:0/2379439865 shutdown_connections 2024-03-29T11:26:06.013 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.022+0000 7f012a4e6640 1 -- 172.21.15.17:0/2379439865 wait complete. 2024-03-29T11:26:06.013 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.022+0000 7f012a4e6640 10 monclient: build_initial_monmap 2024-03-29T11:26:06.013 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.022+0000 7f012a4e6640 10 monclient: monmap: 2024-03-29T11:26:06.013 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:26:06.013 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:06.013 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:26:06.013 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:26:06.013 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:06.014 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:06.014 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:26:06.014 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:26:06.014 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:26:06.014 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:06.014 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.023+0000 7f012a4e6640 1 librados: starting msgr at 2024-03-29T11:26:06.014 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.023+0000 7f012a4e6640 1 librados: starting objecter 2024-03-29T11:26:06.014 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.023+0000 7f012a4e6640 1 Processor -- start 2024-03-29T11:26:06.014 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.023+0000 7f012a4e6640 1 -- start start 2024-03-29T11:26:06.014 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.023+0000 7f012a4e6640 1 librados: setting wanted keys 2024-03-29T11:26:06.014 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.023+0000 7f012a4e6640 1 librados: calling monclient init 2024-03-29T11:26:06.014 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.023+0000 7f012a4e6640 10 monclient: init 2024-03-29T11:26:06.017 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.024+0000 7f012a4e6640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:06.017 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.024+0000 7f012a4e6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:26:06.017 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.024+0000 7f012a4e6640 10 monclient(hunting): picked mon.noname-c con 0x7f01240ac5a0 addr v1:172.21.15.17:6790/0 2024-03-29T11:26:06.017 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.024+0000 7f012a4e6640 10 monclient(hunting): picked mon.noname-a con 0x7f01240b0770 addr v1:172.21.15.17:6789/0 2024-03-29T11:26:06.017 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.027+0000 7f012a4e6640 10 monclient(hunting): picked mon.noname-b con 0x7f01241dfaf0 addr v1:172.21.15.94:6789/0 2024-03-29T11:26:06.017 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.027+0000 7f012a4e6640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0124128650 con 0x7f01240b0770 2024-03-29T11:26:06.017 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.027+0000 7f0128a83640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f01241dfaf0 0x7f01241272e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:47005/0 (socket says 172.21.15.17:47005) 2024-03-29T11:26:06.017 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.027+0000 7f012a4e6640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0124128930 con 0x7f01241dfaf0 2024-03-29T11:26:06.017 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.027+0000 7f0128a83640 1 -- 172.21.15.17:0/1176378900 learned_addr learned my addr 172.21.15.17:0/1176378900 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:26:06.017 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.027+0000 7f012a4e6640 1 -- 172.21.15.17:0/1176378900 --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01241266b0 con 0x7f01240ac5a0 2024-03-29T11:26:06.017 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.027+0000 7f012a4e6640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:06.017 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.027+0000 7f012a4e6640 10 monclient(hunting): authenticate will time out at 1272.232752s 2024-03-29T11:26:06.017 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.028+0000 7f0120ff9640 10 client.?.objecter ms_handle_connect 0x7f01241dfaf0 2024-03-29T11:26:06.018 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.028+0000 7f0120ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:06.018 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.030+0000 7f0120ff9640 10 client.?.objecter ms_handle_connect 0x7f01240ac5a0 2024-03-29T11:26:06.018 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.030+0000 7f0120ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:06.018 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.030+0000 7f0120ff9640 1 -- 172.21.15.17:0/1176378900 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4180317688 0 0) 0x7f01241266b0 con 0x7f01240ac5a0 2024-03-29T11:26:06.018 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.030+0000 7f0120ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:26:06.018 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.030+0000 7f0120ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:06.018 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.031+0000 7f0120ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:06.018 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.031+0000 7f0120ff9640 10 monclient(hunting): my global_id is 4523 2024-03-29T11:26:06.018 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.031+0000 7f0120ff9640 1 -- 172.21.15.17:0/1176378900 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0110004eb0 con 0x7f01240ac5a0 2024-03-29T11:26:06.020 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.034+0000 7f0120ff9640 10 client.?.objecter ms_handle_connect 0x7f01240b0770 2024-03-29T11:26:06.020 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.034+0000 7f0120ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:06.020 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.034+0000 7f0120ff9640 1 -- 172.21.15.17:0/1176378900 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4091240406 0 0) 0x7f0110004eb0 con 0x7f01240ac5a0 2024-03-29T11:26:06.020 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.034+0000 7f0120ff9640 1 -- 172.21.15.17:0/1176378900 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01241266b0 con 0x7f01240ac5a0 2024-03-29T11:26:06.020 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.034+0000 7f0120ff9640 1 -- 172.21.15.17:0/1176378900 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2358835492 0 0) 0x7f0124128650 con 0x7f01240b0770 2024-03-29T11:26:06.020 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.034+0000 7f0120ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:26:06.021 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.034+0000 7f0120ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:06.021 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.034+0000 7f0120ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:06.021 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.034+0000 7f0120ff9640 10 monclient(hunting): my global_id is 4473 2024-03-29T11:26:06.021 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.034+0000 7f0120ff9640 1 -- 172.21.15.17:0/1176378900 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0110004eb0 con 0x7f01240b0770 2024-03-29T11:26:06.021 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.034+0000 7f0120ff9640 1 -- 172.21.15.17:0/1176378900 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1854180252 0 0) 0x7f0124128930 con 0x7f01241dfaf0 2024-03-29T11:26:06.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.034+0000 7f0120ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:26:06.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.034+0000 7f0120ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:06.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.034+0000 7f0120ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:06.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.034+0000 7f0120ff9640 10 monclient(hunting): my global_id is 4507 2024-03-29T11:26:06.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.034+0000 7f0120ff9640 1 -- 172.21.15.17:0/1176378900 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0124128650 con 0x7f01241dfaf0 2024-03-29T11:26:06.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.034+0000 7f0120ff9640 1 -- 172.21.15.17:0/1176378900 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f01140017a0 con 0x7f01240ac5a0 2024-03-29T11:26:06.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.034+0000 7f0120ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:06.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.034+0000 7f0120ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:06.027 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.034+0000 7f0120ff9640 10 monclient(hunting): dump: 2024-03-29T11:26:06.027 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:26:06.027 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:26:06.027 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:06.027 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:06.027 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:06.027 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:06.027 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:26:06.027 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:26:06.027 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:26:06.027 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:06.027 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.034+0000 7f0120ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:06.027 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.035+0000 7f0120ff9640 1 -- 172.21.15.17:0/1176378900 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4075448655 0 0) 0x7f0110004eb0 con 0x7f01240b0770 2024-03-29T11:26:06.027 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.035+0000 7f0120ff9640 1 -- 172.21.15.17:0/1176378900 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0124128930 con 0x7f01240b0770 2024-03-29T11:26:06.027 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.035+0000 7f0120ff9640 1 -- 172.21.15.17:0/1176378900 <== mon.2 v1:172.21.15.17:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1562074641 0 0) 0x7f01241266b0 con 0x7f01240ac5a0 2024-03-29T11:26:06.028 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.035+0000 7f0120ff9640 1 -- 172.21.15.17:0/1176378900 >> v1:172.21.15.94:6789/0 conn(0x7f01241dfaf0 legacy=0x7f01241272e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:06.028 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.035+0000 7f0120ff9640 1 -- 172.21.15.17:0/1176378900 >> v1:172.21.15.17:6789/0 conn(0x7f01240b0770 legacy=0x7f0124126bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:06.028 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.035+0000 7f0120ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:26:06.028 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.035+0000 7f0120ff9640 1 monclient: found mon.c 2024-03-29T11:26:06.028 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.035+0000 7f0120ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:06.028 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.035+0000 7f0120ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:26:06.028 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.035+0000 7f0120ff9640 1 -- 172.21.15.17:0/1176378900 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01241262d0 con 0x7f01240ac5a0 2024-03-29T11:26:06.028 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.035+0000 7f0120ff9640 10 monclient: _finish_auth 0 2024-03-29T11:26:06.028 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.035+0000 7f0120ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:26:06.028 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.035+0000 7f0120ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:06.028 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.035+0000 7f0120ff9640 1 -- 172.21.15.17:0/1176378900 <== mon.2 v1:172.21.15.17:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0114001900 con 0x7f01240ac5a0 2024-03-29T11:26:06.029 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.035+0000 7f0120ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:06.029 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.035+0000 7f0120ff9640 1 -- 172.21.15.17:0/1176378900 <== mon.2 v1:172.21.15.17:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f0114003790 con 0x7f01240ac5a0 2024-03-29T11:26:06.029 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.035+0000 7f0120ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:06.029 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.035+0000 7f0120ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:06.029 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.035+0000 7f0120ff9640 10 monclient: dump: 2024-03-29T11:26:06.029 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:26:06.029 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:26:06.029 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:06.029 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:06.029 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:06.029 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:06.029 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:26:06.029 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:26:06.029 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:26:06.029 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:06.030 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.035+0000 7f012a4e6640 5 monclient: authenticate success, global_id 4523 2024-03-29T11:26:06.030 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.035+0000 7f012a4e6640 10 monclient: _renew_subs 2024-03-29T11:26:06.030 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.035+0000 7f012a4e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:26:06.030 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.035+0000 7f012a4e6640 1 -- 172.21.15.17:0/1176378900 --> v1:172.21.15.17:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01241df0d0 con 0x7f01240ac5a0 2024-03-29T11:26:06.030 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.035+0000 7f012a4e6640 10 client.4523.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:06.030 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.035+0000 7f012a4e6640 10 monclient: _renew_subs 2024-03-29T11:26:06.030 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.035+0000 7f012a4e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:26:06.030 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.035+0000 7f012a4e6640 1 -- 172.21.15.17:0/1176378900 --> v1:172.21.15.17:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0124145950 con 0x7f01240ac5a0 2024-03-29T11:26:06.030 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.035+0000 7f012a4e6640 1 librados: init done 2024-03-29T11:26:06.030 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.035+0000 7f01227fc640 4 set_mon_vals no callback set 2024-03-29T11:26:06.030 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.036+0000 7f010a7fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:06.030 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.036+0000 7f010a7fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:06.030 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.036+0000 7f010a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:26:06.030 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.036+0000 7f010a7fc640 1 -- 172.21.15.17:0/1176378900 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0124126520 con 0x7f01240ac5a0 2024-03-29T11:26:06.031 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.042+0000 7f0120ff9640 1 -- 172.21.15.17:0/1176378900 <== mon.2 v1:172.21.15.17:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f0114019760 con 0x7f01240ac5a0 2024-03-29T11:26:06.031 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.042+0000 7f0120ff9640 1 -- 172.21.15.17:0/1176378900 <== mon.2 v1:172.21.15.17:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f0114083660 con 0x7f01240ac5a0 2024-03-29T11:26:06.031 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.042+0000 7f0120ff9640 10 client.4523.objecter ms_dispatch 0x7f0124000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:06.032 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.042+0000 7f0120ff9640 3 client.4523.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:06.032 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.042+0000 7f0120ff9640 3 client.4523.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:06.032 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.043+0000 7f0120ff9640 20 client.4523.objecter dump_active .. 0 homeless 2024-03-29T11:26:06.032 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.043+0000 7f01237fe640 10 monclient: get_auth_request con 0x7f011006e060 auth_method 0 2024-03-29T11:26:06.032 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.045+0000 7f0120ff9640 10 client.4523.objecter ms_handle_connect 0x7f011006e060 2024-03-29T11:26:06.033 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.048+0000 7f0120ff9640 1 -- 172.21.15.17:0/1176378900 <== mon.2 v1:172.21.15.17:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f011404ef40 con 0x7f01240ac5a0 2024-03-29T11:26:06.033 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.048+0000 7f0120ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:06.033 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.048+0000 7f0120ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:06.366 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.380+0000 7f010a7fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:06.366 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.380+0000 7f010a7fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:06.366 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.380+0000 7f010a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:26:06.366 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.380+0000 7f010a7fc640 1 -- 172.21.15.17:0/1176378900 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01240ab560 con 0x7f01240ac5a0 2024-03-29T11:26:06.370 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.385+0000 7f0120ff9640 1 -- 172.21.15.17:0/1176378900 <== mon.2 v1:172.21.15.17:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f0114050e70 con 0x7f01240ac5a0 2024-03-29T11:26:06.371 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.385+0000 7f0120ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:06.371 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.385+0000 7f0120ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:06.371 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:26:06.371 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:06.375 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.389+0000 7f010a7fc640 10 librados: watch_flush enter 2024-03-29T11:26:06.375 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.389+0000 7f010a7fc640 10 librados: watch_flush exit 2024-03-29T11:26:06.375 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.389+0000 7f010a7fc640 20 client.4523.objecter shutdown clearing up homeless session... 2024-03-29T11:26:06.375 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.389+0000 7f010a7fc640 10 client.4523.objecter successfully canceled tick 2024-03-29T11:26:06.375 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.389+0000 7f010a7fc640 1 -- 172.21.15.17:0/1176378900 >> v1:172.21.15.17:6816/3236185052 conn(0x7f011006e060 legacy=0x7f0110070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:06.375 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.389+0000 7f010a7fc640 10 monclient: shutdown 2024-03-29T11:26:06.375 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.389+0000 7f010a7fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:06.375 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.389+0000 7f010a7fc640 1 -- 172.21.15.17:0/1176378900 >> v1:172.21.15.17:6790/0 conn(0x7f01240ac5a0 legacy=0x7f0124145240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:06.375 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.389+0000 7f010a7fc640 1 -- 172.21.15.17:0/1176378900 shutdown_connections 2024-03-29T11:26:06.376 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.389+0000 7f010a7fc640 1 -- 172.21.15.17:0/1176378900 >> 172.21.15.17:0/1176378900 conn(0x7f012406b660 msgr2=0x7f01240b3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:06.376 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.390+0000 7f010a7fc640 1 -- 172.21.15.17:0/1176378900 shutdown_connections 2024-03-29T11:26:06.376 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.390+0000 7f010a7fc640 1 -- 172.21.15.17:0/1176378900 wait complete. 2024-03-29T11:26:06.376 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:06.390+0000 7f010a7fc640 1 librados: shutdown 2024-03-29T11:26:06.395 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:13.202 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:13.353 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.367+0000 7feffdfcf640 10 monclient: get_monmap_and_config 2024-03-29T11:26:13.353 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.367+0000 7feffdfcf640 10 monclient: build_initial_monmap 2024-03-29T11:26:13.353 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.367+0000 7feffdfcf640 10 monclient: monmap: 2024-03-29T11:26:13.353 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:26:13.353 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:26:13.353 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:26:13.368567+0000 2024-03-29T11:26:13.353 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:26:13.368567+0000 2024-03-29T11:26:13.353 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:13.354 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:13.354 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:26:13.354 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:26:13.354 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:26:13.354 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:13.360 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.372+0000 7feffdfcf640 1 Processor -- start 2024-03-29T11:26:13.360 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.373+0000 7feffdfcf640 1 -- start start 2024-03-29T11:26:13.360 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.373+0000 7feffdfcf640 10 monclient: init 2024-03-29T11:26:13.360 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.373+0000 7feffdfcf640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:13.360 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.373+0000 7feffdfcf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:26:13.360 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.374+0000 7feffdfcf640 10 monclient(hunting): picked mon.noname-b con 0x7feff80b1f60 addr v1:172.21.15.17:6790/0 2024-03-29T11:26:13.360 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.374+0000 7feffdfcf640 10 monclient(hunting): picked mon.noname-c con 0x7feff8071eb0 addr v1:172.21.15.94:6789/0 2024-03-29T11:26:13.360 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.374+0000 7feffdfcf640 10 monclient(hunting): picked mon.noname-a con 0x7feff8143d10 addr v1:172.21.15.17:6789/0 2024-03-29T11:26:13.360 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.374+0000 7feffdfcf640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feff806ef80 con 0x7feff8143d10 2024-03-29T11:26:13.360 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.374+0000 7feffdfcf640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feff806f260 con 0x7feff8071eb0 2024-03-29T11:26:13.361 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.374+0000 7feffdfcf640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feff80b40c0 con 0x7feff80b1f60 2024-03-29T11:26:13.361 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.374+0000 7feffdfcf640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:13.361 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.374+0000 7feffdfcf640 10 monclient(hunting): authenticate will time out at 1279.578948s 2024-03-29T11:26:13.363 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.377+0000 7feff77fe640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7feff80b1f60 0x7feff8143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:37707/0 (socket says 172.21.15.17:37707) 2024-03-29T11:26:13.364 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.377+0000 7feff77fe640 1 -- 172.21.15.17:0/3809590872 learned_addr learned my addr 172.21.15.17:0/3809590872 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:26:13.364 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.377+0000 7feff67fc640 1 -- 172.21.15.17:0/3809590872 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2434985421 0 0) 0x7feff80b40c0 con 0x7feff80b1f60 2024-03-29T11:26:13.364 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.377+0000 7feff67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:13.364 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.377+0000 7feff67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:13.364 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.377+0000 7feff67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:13.364 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.377+0000 7feff67fc640 10 monclient(hunting): my global_id is 4526 2024-03-29T11:26:13.364 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.377+0000 7feff67fc640 1 -- 172.21.15.17:0/3809590872 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefe0003520 con 0x7feff80b1f60 2024-03-29T11:26:13.364 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.378+0000 7feff67fc640 1 -- 172.21.15.17:0/3809590872 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 61342984 0 0) 0x7fefe0003520 con 0x7feff80b1f60 2024-03-29T11:26:13.364 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.378+0000 7feff67fc640 1 -- 172.21.15.17:0/3809590872 >> v1:172.21.15.94:6789/0 conn(0x7feff8071eb0 legacy=0x7feff8146470 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:26:13.364 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.378+0000 7feff67fc640 1 -- 172.21.15.17:0/3809590872 >> v1:172.21.15.17:6789/0 conn(0x7feff8143d10 legacy=0x7feff8146b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:13.365 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.380+0000 7feff67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:26:13.365 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.380+0000 7feff67fc640 1 monclient: found mon.noname-b 2024-03-29T11:26:13.365 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.380+0000 7feff67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:13.365 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.380+0000 7feff67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.17:6790/0 2024-03-29T11:26:13.365 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.380+0000 7feff67fc640 1 -- 172.21.15.17:0/3809590872 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff8155d70 con 0x7feff80b1f60 2024-03-29T11:26:13.365 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.380+0000 7feff67fc640 10 monclient: _finish_auth 0 2024-03-29T11:26:13.366 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.380+0000 7feff67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:26:13.366 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.380+0000 7feff67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:13.366 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.380+0000 7feff67fc640 1 -- 172.21.15.17:0/3809590872 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7feff0001a10 con 0x7feff80b1f60 2024-03-29T11:26:13.366 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.380+0000 7feff67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:13.366 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.380+0000 7feff67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:13.366 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.380+0000 7feff67fc640 10 monclient: dump: 2024-03-29T11:26:13.366 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:26:13.366 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:26:13.366 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:13.366 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:13.366 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:13.366 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:13.366 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:26:13.366 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:26:13.367 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:26:13.367 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:13.367 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.380+0000 7feffdfcf640 5 monclient: authenticate success, global_id 4526 2024-03-29T11:26:13.367 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.380+0000 7feffdfcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:13.369 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.384+0000 7feff67fc640 1 -- 172.21.15.17:0/3809590872 <== mon.2 v1:172.21.15.17:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feff0001be0 con 0x7feff80b1f60 2024-03-29T11:26:13.369 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.384+0000 7feff67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:13.369 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.384+0000 7feff67fc640 1 -- 172.21.15.17:0/3809590872 <== mon.2 v1:172.21.15.17:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7feff0002030 con 0x7feff80b1f60 2024-03-29T11:26:13.370 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.384+0000 7feff67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:13.370 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.384+0000 7feff67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:13.370 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.384+0000 7feff67fc640 10 monclient: dump: 2024-03-29T11:26:13.370 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:26:13.370 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:26:13.370 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:13.370 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:13.370 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:13.370 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:13.370 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:26:13.370 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:26:13.370 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:26:13.370 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:13.370 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.384+0000 7feffdfcf640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:13.370 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.384+0000 7feffdfcf640 4 set_mon_vals no callback set 2024-03-29T11:26:13.371 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.385+0000 7feffdfcf640 10 monclient: shutdown 2024-03-29T11:26:13.371 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.385+0000 7feffdfcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:13.371 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.385+0000 7feffdfcf640 1 -- 172.21.15.17:0/3809590872 >> v1:172.21.15.17:6790/0 conn(0x7feff80b1f60 legacy=0x7feff8143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:13.371 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.385+0000 7feffdfcf640 1 -- 172.21.15.17:0/3809590872 shutdown_connections 2024-03-29T11:26:13.371 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.385+0000 7feffdfcf640 1 -- 172.21.15.17:0/3809590872 >> 172.21.15.17:0/3809590872 conn(0x7feff8066a00 msgr2=0x7feff8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:13.375 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.386+0000 7feffdfcf640 1 -- 172.21.15.17:0/3809590872 shutdown_connections 2024-03-29T11:26:13.375 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.387+0000 7feffdfcf640 1 -- 172.21.15.17:0/3809590872 wait complete. 2024-03-29T11:26:13.375 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.387+0000 7feffdfcf640 10 monclient: build_initial_monmap 2024-03-29T11:26:13.375 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.387+0000 7feffdfcf640 10 monclient: monmap: 2024-03-29T11:26:13.375 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:26:13.375 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:13.375 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:26:13.376 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:26:13.376 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:13.376 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:13.376 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:26:13.376 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:26:13.376 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:26:13.376 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:13.376 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.388+0000 7feffdfcf640 1 librados: starting msgr at 2024-03-29T11:26:13.376 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.388+0000 7feffdfcf640 1 librados: starting objecter 2024-03-29T11:26:13.376 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.388+0000 7feffdfcf640 1 Processor -- start 2024-03-29T11:26:13.380 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.391+0000 7feffdfcf640 1 -- start start 2024-03-29T11:26:13.380 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.391+0000 7feffdfcf640 1 librados: setting wanted keys 2024-03-29T11:26:13.380 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.391+0000 7feffdfcf640 1 librados: calling monclient init 2024-03-29T11:26:13.380 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.391+0000 7feffdfcf640 10 monclient: init 2024-03-29T11:26:13.380 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.392+0000 7feffdfcf640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:13.380 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.392+0000 7feffdfcf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:26:13.380 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.392+0000 7feffdfcf640 10 monclient(hunting): picked mon.noname-a con 0x7feff80b1f60 addr v1:172.21.15.17:6789/0 2024-03-29T11:26:13.380 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.392+0000 7feffdfcf640 10 monclient(hunting): picked mon.noname-b con 0x7feff8143d10 addr v1:172.21.15.94:6789/0 2024-03-29T11:26:13.380 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.392+0000 7feffdfcf640 10 monclient(hunting): picked mon.noname-c con 0x7feff806f620 addr v1:172.21.15.17:6790/0 2024-03-29T11:26:13.380 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.392+0000 7feffdfcf640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feff81328d0 con 0x7feff80b1f60 2024-03-29T11:26:13.380 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.392+0000 7feffdfcf640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feff80703a0 con 0x7feff8143d10 2024-03-29T11:26:13.380 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.392+0000 7feffdfcf640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feff8070680 con 0x7feff806f620 2024-03-29T11:26:13.381 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.392+0000 7feffdfcf640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:13.381 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.392+0000 7feffdfcf640 10 monclient(hunting): authenticate will time out at 1279.597133s 2024-03-29T11:26:13.381 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.396+0000 7feff77fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7feff80b1f60 0x7feff8071010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:40095/0 (socket says 172.21.15.17:40095) 2024-03-29T11:26:13.381 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.396+0000 7feff77fe640 1 -- 172.21.15.17:0/483574326 learned_addr learned my addr 172.21.15.17:0/483574326 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:26:13.381 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.396+0000 7fefdffff640 10 client.?.objecter ms_handle_connect 0x7feff80b1f60 2024-03-29T11:26:13.382 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.396+0000 7fefdffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:13.383 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.397+0000 7fefdffff640 1 -- 172.21.15.17:0/483574326 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2949914674 0 0) 0x7feff81328d0 con 0x7feff80b1f60 2024-03-29T11:26:13.383 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.397+0000 7fefdffff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:13.383 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.397+0000 7fefdffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:13.383 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.397+0000 7fefdffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:13.383 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.397+0000 7fefdffff640 10 monclient(hunting): my global_id is 4476 2024-03-29T11:26:13.383 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.397+0000 7fefdffff640 1 -- 172.21.15.17:0/483574326 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefe4004eb0 con 0x7feff80b1f60 2024-03-29T11:26:13.384 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.398+0000 7fefdffff640 1 -- 172.21.15.17:0/483574326 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3298382731 0 0) 0x7fefe4004eb0 con 0x7feff80b1f60 2024-03-29T11:26:13.384 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.398+0000 7fefdffff640 1 -- 172.21.15.17:0/483574326 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feff81328d0 con 0x7feff80b1f60 2024-03-29T11:26:13.384 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.398+0000 7fefdffff640 1 -- 172.21.15.17:0/483574326 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7feff0001ef0 con 0x7feff80b1f60 2024-03-29T11:26:13.384 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.398+0000 7fefdffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:13.384 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.398+0000 7fefdffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:13.384 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.398+0000 7fefdffff640 10 monclient(hunting): dump: 2024-03-29T11:26:13.384 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:26:13.384 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:26:13.384 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:13.384 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:13.385 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:13.385 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:13.385 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:26:13.385 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:26:13.385 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:26:13.385 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:13.385 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.398+0000 7fefdffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:13.387 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.399+0000 7fefdffff640 1 -- 172.21.15.17:0/483574326 <== mon.0 v1:172.21.15.17:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1290919441 0 0) 0x7feff81328d0 con 0x7feff80b1f60 2024-03-29T11:26:13.387 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.399+0000 7fefdffff640 1 -- 172.21.15.17:0/483574326 >> v1:172.21.15.17:6790/0 conn(0x7feff806f620 legacy=0x7feff81321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:13.387 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.399+0000 7fefdffff640 1 -- 172.21.15.17:0/483574326 >> v1:172.21.15.94:6789/0 conn(0x7feff8143d10 legacy=0x7feff8131ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:13.387 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.399+0000 7fefdffff640 10 monclient: _finish_hunting 0 2024-03-29T11:26:13.387 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.399+0000 7fefdffff640 1 monclient: found mon.a 2024-03-29T11:26:13.387 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.399+0000 7fefdffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:13.387 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.399+0000 7fefdffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:26:13.387 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.399+0000 7fefdffff640 1 -- 172.21.15.17:0/483574326 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff8070960 con 0x7feff80b1f60 2024-03-29T11:26:13.387 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.399+0000 7fefdffff640 10 monclient: _finish_auth 0 2024-03-29T11:26:13.387 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.399+0000 7fefdffff640 10 monclient: _check_auth_tickets 2024-03-29T11:26:13.388 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.399+0000 7fefdffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:13.388 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.399+0000 7feffdfcf640 5 monclient: authenticate success, global_id 4476 2024-03-29T11:26:13.388 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.399+0000 7feffdfcf640 10 monclient: _renew_subs 2024-03-29T11:26:13.388 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.399+0000 7feffdfcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:26:13.388 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.399+0000 7feffdfcf640 1 -- 172.21.15.17:0/483574326 --> v1:172.21.15.17:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feff81544d0 con 0x7feff80b1f60 2024-03-29T11:26:13.388 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.399+0000 7feffdfcf640 10 client.4476.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:13.388 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.399+0000 7feffdfcf640 10 monclient: _renew_subs 2024-03-29T11:26:13.388 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.399+0000 7feffdfcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:26:13.388 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.399+0000 7feffdfcf640 1 -- 172.21.15.17:0/483574326 --> v1:172.21.15.17:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feff8155220 con 0x7feff80b1f60 2024-03-29T11:26:13.388 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.399+0000 7feffdfcf640 1 librados: init done 2024-03-29T11:26:13.388 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.402+0000 7fefdffff640 1 -- 172.21.15.17:0/483574326 <== mon.0 v1:172.21.15.17:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feff0004f70 con 0x7feff80b1f60 2024-03-29T11:26:13.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.402+0000 7fefdffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:13.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.402+0000 7fefdffff640 1 -- 172.21.15.17:0/483574326 <== mon.0 v1:172.21.15.17:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7feff00053c0 con 0x7feff80b1f60 2024-03-29T11:26:13.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.402+0000 7fefdffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:13.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.402+0000 7fefdffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:13.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.402+0000 7fefdffff640 10 monclient: dump: 2024-03-29T11:26:13.389 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:26:13.389 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:26:13.389 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:13.389 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:13.389 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:13.389 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:13.389 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:26:13.389 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:26:13.389 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:26:13.390 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:13.390 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.402+0000 7fefdffff640 1 -- 172.21.15.17:0/483574326 <== mon.0 v1:172.21.15.17:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7feff00055e0 con 0x7feff80b1f60 2024-03-29T11:26:13.390 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.402+0000 7feff5ffb640 4 set_mon_vals no callback set 2024-03-29T11:26:13.390 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.403+0000 7fefdffff640 1 -- 172.21.15.17:0/483574326 <== mon.0 v1:172.21.15.17:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7feff00851d0 con 0x7feff80b1f60 2024-03-29T11:26:13.390 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.403+0000 7fefdffff640 10 client.4476.objecter ms_dispatch 0x7feff8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:13.390 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.403+0000 7fefdffff640 3 client.4476.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:13.391 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.403+0000 7fefdffff640 3 client.4476.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:13.391 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.403+0000 7fefdffff640 20 client.4476.objecter dump_active .. 0 homeless 2024-03-29T11:26:13.391 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.403+0000 7feffdfcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:13.391 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.403+0000 7feffdfcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:13.391 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.403+0000 7feffdfcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:26:13.391 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.403+0000 7feffdfcf640 1 -- 172.21.15.17:0/483574326 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefd8005080 con 0x7feff80b1f60 2024-03-29T11:26:13.391 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.404+0000 7feff6ffd640 10 monclient: get_auth_request con 0x7fefe406ddf0 auth_method 0 2024-03-29T11:26:13.395 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.405+0000 7fefdffff640 10 client.4476.objecter ms_handle_connect 0x7fefe406ddf0 2024-03-29T11:26:13.398 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.412+0000 7fefdffff640 1 -- 172.21.15.17:0/483574326 <== mon.0 v1:172.21.15.17:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feff0050b30 con 0x7feff80b1f60 2024-03-29T11:26:13.398 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.412+0000 7fefdffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:13.398 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.412+0000 7fefdffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:13.731 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.745+0000 7feffdfcf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:13.731 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.745+0000 7feffdfcf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:13.731 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.745+0000 7feffdfcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:26:13.731 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.745+0000 7feffdfcf640 1 -- 172.21.15.17:0/483574326 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fefd8005330 con 0x7feff80b1f60 2024-03-29T11:26:13.732 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.747+0000 7fefdffff640 1 -- 172.21.15.17:0/483574326 <== mon.0 v1:172.21.15.17:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7feff0052a60 con 0x7feff80b1f60 2024-03-29T11:26:13.732 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.747+0000 7fefdffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:13.732 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.747+0000 7fefdffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:13.735 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:26:13.735 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:13.743 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.753+0000 7feffdfcf640 10 librados: watch_flush enter 2024-03-29T11:26:13.743 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.753+0000 7feffdfcf640 10 librados: watch_flush exit 2024-03-29T11:26:13.743 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.753+0000 7feffdfcf640 20 client.4476.objecter shutdown clearing up homeless session... 2024-03-29T11:26:13.743 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.753+0000 7feffdfcf640 10 client.4476.objecter successfully canceled tick 2024-03-29T11:26:13.743 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.755+0000 7feffdfcf640 1 -- 172.21.15.17:0/483574326 >> v1:172.21.15.17:6816/3236185052 conn(0x7fefe406ddf0 legacy=0x7fefe40702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:13.743 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.755+0000 7feffdfcf640 10 monclient: shutdown 2024-03-29T11:26:13.743 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.755+0000 7feffdfcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:13.743 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.755+0000 7feffdfcf640 1 -- 172.21.15.17:0/483574326 >> v1:172.21.15.17:6789/0 conn(0x7feff80b1f60 legacy=0x7feff8071010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:13.743 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.755+0000 7feffdfcf640 1 -- 172.21.15.17:0/483574326 shutdown_connections 2024-03-29T11:26:13.743 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.755+0000 7feffdfcf640 1 -- 172.21.15.17:0/483574326 >> 172.21.15.17:0/483574326 conn(0x7feff8066a00 msgr2=0x7feff80b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:13.744 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.756+0000 7feffdfcf640 1 -- 172.21.15.17:0/483574326 shutdown_connections 2024-03-29T11:26:13.744 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.756+0000 7feffdfcf640 1 -- 172.21.15.17:0/483574326 wait complete. 2024-03-29T11:26:13.744 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:13.757+0000 7feffdfcf640 1 librados: shutdown 2024-03-29T11:26:13.759 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:20.567 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:20.718 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.732+0000 7f8d6aa2b640 10 monclient: get_monmap_and_config 2024-03-29T11:26:20.719 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.732+0000 7f8d6aa2b640 10 monclient: build_initial_monmap 2024-03-29T11:26:20.719 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.732+0000 7f8d6aa2b640 10 monclient: monmap: 2024-03-29T11:26:20.719 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:26:20.719 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:26:20.719 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:26:20.733460+0000 2024-03-29T11:26:20.719 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:26:20.733460+0000 2024-03-29T11:26:20.719 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:20.719 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:20.719 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:26:20.719 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:26:20.719 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:26:20.719 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:20.720 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.733+0000 7f8d6aa2b640 1 Processor -- start 2024-03-29T11:26:20.720 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.733+0000 7f8d6aa2b640 1 -- start start 2024-03-29T11:26:20.720 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.733+0000 7f8d6aa2b640 10 monclient: init 2024-03-29T11:26:20.720 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.734+0000 7f8d6aa2b640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:20.720 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.734+0000 7f8d6aa2b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:26:20.721 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.734+0000 7f8d6aa2b640 10 monclient(hunting): picked mon.noname-c con 0x7f8d640a9750 addr v1:172.21.15.94:6789/0 2024-03-29T11:26:20.721 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.735+0000 7f8d6aa2b640 10 monclient(hunting): picked mon.noname-a con 0x7f8d640ae220 addr v1:172.21.15.17:6789/0 2024-03-29T11:26:20.721 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.735+0000 7f8d6aa2b640 10 monclient(hunting): picked mon.noname-b con 0x7f8d6413b9d0 addr v1:172.21.15.17:6790/0 2024-03-29T11:26:20.721 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.735+0000 7f8d6aa2b640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d64142490 con 0x7f8d640ae220 2024-03-29T11:26:20.722 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.735+0000 7f8d6aa2b640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d640ac130 con 0x7f8d640a9750 2024-03-29T11:26:20.722 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.735+0000 7f8d6aa2b640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d640ac410 con 0x7f8d6413b9d0 2024-03-29T11:26:20.722 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.735+0000 7f8d6aa2b640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:20.722 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.735+0000 7f8d6aa2b640 10 monclient(hunting): authenticate will time out at 1286.940453s 2024-03-29T11:26:20.723 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.738+0000 7f8d69228640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f8d640ae220 0x7f8d6413b2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:52501/0 (socket says 172.21.15.17:52501) 2024-03-29T11:26:20.724 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.738+0000 7f8d69228640 1 -- 172.21.15.17:0/440586170 learned_addr learned my addr 172.21.15.17:0/440586170 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:26:20.725 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.739+0000 7f8d68a27640 1 -- 172.21.15.17:0/440586170 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3509357981 0 0) 0x7f8d64142490 con 0x7f8d640ae220 2024-03-29T11:26:20.725 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.739+0000 7f8d68a27640 10 monclient(hunting): _negotiate 2024-03-29T11:26:20.725 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.739+0000 7f8d68a27640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:20.725 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.739+0000 7f8d68a27640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:20.725 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.739+0000 7f8d68a27640 10 monclient(hunting): my global_id is 4479 2024-03-29T11:26:20.725 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.739+0000 7f8d68a27640 1 -- 172.21.15.17:0/440586170 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d50003520 con 0x7f8d640ae220 2024-03-29T11:26:20.726 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.740+0000 7f8d68a27640 1 -- 172.21.15.17:0/440586170 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 895948242 0 0) 0x7f8d640ac130 con 0x7f8d640a9750 2024-03-29T11:26:20.726 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.740+0000 7f8d68a27640 10 monclient(hunting): _negotiate 2024-03-29T11:26:20.726 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.740+0000 7f8d68a27640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:20.726 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.740+0000 7f8d68a27640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:20.726 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.740+0000 7f8d68a27640 10 monclient(hunting): my global_id is 4510 2024-03-29T11:26:20.726 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.740+0000 7f8d68a27640 1 -- 172.21.15.17:0/440586170 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d64142490 con 0x7f8d640a9750 2024-03-29T11:26:20.727 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.740+0000 7f8d68a27640 1 -- 172.21.15.17:0/440586170 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 31506308 0 0) 0x7f8d50003520 con 0x7f8d640ae220 2024-03-29T11:26:20.727 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.740+0000 7f8d68a27640 1 -- 172.21.15.17:0/440586170 >> v1:172.21.15.17:6790/0 conn(0x7f8d6413b9d0 legacy=0x7f8d64149930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:20.727 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.740+0000 7f8d68a27640 1 -- 172.21.15.17:0/440586170 >> v1:172.21.15.94:6789/0 conn(0x7f8d640a9750 legacy=0x7f8d64141d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:20.727 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.740+0000 7f8d68a27640 10 monclient: _finish_hunting 0 2024-03-29T11:26:20.727 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.740+0000 7f8d68a27640 1 monclient: found mon.noname-a 2024-03-29T11:26:20.728 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.740+0000 7f8d68a27640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:20.728 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.740+0000 7f8d68a27640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.17:6789/0 2024-03-29T11:26:20.728 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.741+0000 7f8d68a27640 1 -- 172.21.15.17:0/440586170 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d640afa40 con 0x7f8d640ae220 2024-03-29T11:26:20.728 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.741+0000 7f8d68a27640 10 monclient: _finish_auth 0 2024-03-29T11:26:20.728 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.741+0000 7f8d68a27640 10 monclient: _check_auth_tickets 2024-03-29T11:26:20.728 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.741+0000 7f8d68a27640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:20.728 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.741+0000 7f8d68a27640 1 -- 172.21.15.17:0/440586170 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f8d60001690 con 0x7f8d640ae220 2024-03-29T11:26:20.728 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.741+0000 7f8d68a27640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:20.728 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.741+0000 7f8d68a27640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:20.728 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.741+0000 7f8d68a27640 10 monclient: dump: 2024-03-29T11:26:20.728 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:26:20.728 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:26:20.728 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:20.728 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:20.728 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:20.729 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:20.729 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:26:20.729 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:26:20.729 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:26:20.729 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:20.729 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.741+0000 7f8d6aa2b640 5 monclient: authenticate success, global_id 4479 2024-03-29T11:26:20.729 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.741+0000 7f8d6aa2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:20.729 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.741+0000 7f8d68a27640 1 -- 172.21.15.17:0/440586170 <== mon.0 v1:172.21.15.17:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d600018b0 con 0x7f8d640ae220 2024-03-29T11:26:20.729 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.741+0000 7f8d68a27640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:20.729 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.741+0000 7f8d68a27640 1 -- 172.21.15.17:0/440586170 <== mon.0 v1:172.21.15.17:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f8d60001bc0 con 0x7f8d640ae220 2024-03-29T11:26:20.729 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.741+0000 7f8d68a27640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:20.729 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.741+0000 7f8d68a27640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:20.729 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.741+0000 7f8d68a27640 10 monclient: dump: 2024-03-29T11:26:20.729 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:26:20.729 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:26:20.730 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:20.730 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:20.730 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:20.730 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:20.730 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:26:20.730 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:26:20.730 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:26:20.730 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:20.730 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.741+0000 7f8d6aa2b640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:20.730 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.741+0000 7f8d6aa2b640 4 set_mon_vals no callback set 2024-03-29T11:26:20.732 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.742+0000 7f8d6aa2b640 10 monclient: shutdown 2024-03-29T11:26:20.732 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.742+0000 7f8d6aa2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:20.732 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.742+0000 7f8d6aa2b640 1 -- 172.21.15.17:0/440586170 >> v1:172.21.15.17:6789/0 conn(0x7f8d640ae220 legacy=0x7f8d6413b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:20.732 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.743+0000 7f8d6aa2b640 1 -- 172.21.15.17:0/440586170 shutdown_connections 2024-03-29T11:26:20.732 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.743+0000 7f8d6aa2b640 1 -- 172.21.15.17:0/440586170 >> 172.21.15.17:0/440586170 conn(0x7f8d640673e0 msgr2=0x7f8d640b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:20.732 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.743+0000 7f8d6aa2b640 1 -- 172.21.15.17:0/440586170 shutdown_connections 2024-03-29T11:26:20.732 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.744+0000 7f8d6aa2b640 1 -- 172.21.15.17:0/440586170 wait complete. 2024-03-29T11:26:20.732 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.745+0000 7f8d6aa2b640 10 monclient: build_initial_monmap 2024-03-29T11:26:20.732 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.745+0000 7f8d6aa2b640 10 monclient: monmap: 2024-03-29T11:26:20.732 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:26:20.732 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:20.732 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:26:20.732 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:26:20.732 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:20.733 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:20.733 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:26:20.733 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:26:20.733 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:26:20.733 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:20.733 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.745+0000 7f8d6aa2b640 1 librados: starting msgr at 2024-03-29T11:26:20.733 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.745+0000 7f8d6aa2b640 1 librados: starting objecter 2024-03-29T11:26:20.733 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.746+0000 7f8d6aa2b640 1 Processor -- start 2024-03-29T11:26:20.733 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.747+0000 7f8d6aa2b640 1 -- start start 2024-03-29T11:26:20.734 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.747+0000 7f8d6aa2b640 1 librados: setting wanted keys 2024-03-29T11:26:20.734 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.747+0000 7f8d6aa2b640 1 librados: calling monclient init 2024-03-29T11:26:20.734 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.747+0000 7f8d6aa2b640 10 monclient: init 2024-03-29T11:26:20.734 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.748+0000 7f8d6aa2b640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:20.734 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.748+0000 7f8d6aa2b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:26:20.739 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.750+0000 7f8d6aa2b640 10 monclient(hunting): picked mon.noname-c con 0x7f8d640a9750 addr v1:172.21.15.17:6790/0 2024-03-29T11:26:20.739 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.750+0000 7f8d6aa2b640 10 monclient(hunting): picked mon.noname-b con 0x7f8d640ae220 addr v1:172.21.15.94:6789/0 2024-03-29T11:26:20.739 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.750+0000 7f8d6aa2b640 10 monclient(hunting): picked mon.noname-a con 0x7f8d6413b9d0 addr v1:172.21.15.17:6789/0 2024-03-29T11:26:20.739 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.750+0000 7f8d6aa2b640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d64140180 con 0x7f8d6413b9d0 2024-03-29T11:26:20.739 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.750+0000 7f8d6aa2b640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d64135bb0 con 0x7f8d640ae220 2024-03-29T11:26:20.740 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.751+0000 7f8d6aa2b640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d641358d0 con 0x7f8d640a9750 2024-03-29T11:26:20.740 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.751+0000 7f8d6aa2b640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:20.740 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.751+0000 7f8d6aa2b640 10 monclient(hunting): authenticate will time out at 1286.955910s 2024-03-29T11:26:20.740 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.753+0000 7f8d69a29640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f8d640a9750 0x7f8d64137330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:42069/0 (socket says 172.21.15.17:42069) 2024-03-29T11:26:20.740 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.753+0000 7f8d69a29640 1 -- 172.21.15.17:0/3062988222 learned_addr learned my addr 172.21.15.17:0/3062988222 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:26:20.740 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.753+0000 7f8d5a7fc640 10 client.?.objecter ms_handle_connect 0x7f8d640a9750 2024-03-29T11:26:20.740 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.753+0000 7f8d5a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:20.741 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.753+0000 7f8d5a7fc640 1 -- 172.21.15.17:0/3062988222 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1352280799 0 0) 0x7f8d641358d0 con 0x7f8d640a9750 2024-03-29T11:26:20.741 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.753+0000 7f8d5a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:20.741 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.753+0000 7f8d5a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:20.741 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.753+0000 7f8d5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:20.741 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.753+0000 7f8d5a7fc640 10 monclient(hunting): my global_id is 4532 2024-03-29T11:26:20.741 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.753+0000 7f8d5a7fc640 1 -- 172.21.15.17:0/3062988222 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d4800d6c0 con 0x7f8d640a9750 2024-03-29T11:26:20.745 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.755+0000 7f8d5a7fc640 10 client.?.objecter ms_handle_connect 0x7f8d640ae220 2024-03-29T11:26:20.745 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.755+0000 7f8d5a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:20.745 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.755+0000 7f8d5a7fc640 1 -- 172.21.15.17:0/3062988222 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3071658500 0 0) 0x7f8d64135bb0 con 0x7f8d640ae220 2024-03-29T11:26:20.745 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.755+0000 7f8d5a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:20.745 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.755+0000 7f8d5a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:20.746 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.755+0000 7f8d5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:20.746 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.755+0000 7f8d5a7fc640 10 monclient(hunting): my global_id is 4513 2024-03-29T11:26:20.746 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.756+0000 7f8d5a7fc640 1 -- 172.21.15.17:0/3062988222 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d641358d0 con 0x7f8d640ae220 2024-03-29T11:26:20.746 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.756+0000 7f8d5a7fc640 1 -- 172.21.15.17:0/3062988222 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4124232256 0 0) 0x7f8d4800d6c0 con 0x7f8d640a9750 2024-03-29T11:26:20.746 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.756+0000 7f8d5a7fc640 1 -- 172.21.15.17:0/3062988222 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d64135bb0 con 0x7f8d640a9750 2024-03-29T11:26:20.746 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.756+0000 7f8d5a7fc640 1 -- 172.21.15.17:0/3062988222 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f8d5c001d90 con 0x7f8d640a9750 2024-03-29T11:26:20.746 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.756+0000 7f8d5a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:20.746 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.756+0000 7f8d5a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:20.746 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.756+0000 7f8d5a7fc640 10 monclient(hunting): dump: 2024-03-29T11:26:20.746 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:26:20.746 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:26:20.746 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:20.746 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:20.746 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:20.747 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:20.747 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:26:20.747 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:26:20.747 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:26:20.747 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:20.747 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.756+0000 7f8d5a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:20.747 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.756+0000 7f8d5a7fc640 1 -- 172.21.15.17:0/3062988222 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 385519882 0 0) 0x7f8d641358d0 con 0x7f8d640ae220 2024-03-29T11:26:20.747 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.756+0000 7f8d5a7fc640 1 -- 172.21.15.17:0/3062988222 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d4800d6c0 con 0x7f8d640ae220 2024-03-29T11:26:20.747 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.756+0000 7f8d5a7fc640 1 -- 172.21.15.17:0/3062988222 <== mon.2 v1:172.21.15.17:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 800367876 0 0) 0x7f8d64135bb0 con 0x7f8d640a9750 2024-03-29T11:26:20.747 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.756+0000 7f8d5a7fc640 1 -- 172.21.15.17:0/3062988222 >> v1:172.21.15.94:6789/0 conn(0x7f8d640ae220 legacy=0x7f8d641d3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:20.747 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.756+0000 7f8d5a7fc640 1 -- 172.21.15.17:0/3062988222 >> v1:172.21.15.17:6789/0 conn(0x7f8d6413b9d0 legacy=0x7f8d641de150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:20.747 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.757+0000 7f8d5a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:26:20.747 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.757+0000 7f8d5a7fc640 1 monclient: found mon.c 2024-03-29T11:26:20.747 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.757+0000 7f8d5a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:20.747 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.757+0000 7f8d5a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:26:20.748 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.757+0000 7f8d5a7fc640 1 -- 172.21.15.17:0/3062988222 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d64137a40 con 0x7f8d640a9750 2024-03-29T11:26:20.748 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.757+0000 7f8d5a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:26:20.748 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.757+0000 7f8d5a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:26:20.748 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.757+0000 7f8d5a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:20.748 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.757+0000 7f8d6aa2b640 5 monclient: authenticate success, global_id 4532 2024-03-29T11:26:20.748 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.757+0000 7f8d6aa2b640 10 monclient: _renew_subs 2024-03-29T11:26:20.748 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.757+0000 7f8d5a7fc640 1 -- 172.21.15.17:0/3062988222 <== mon.2 v1:172.21.15.17:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d5c001ef0 con 0x7f8d640a9750 2024-03-29T11:26:20.748 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.757+0000 7f8d6aa2b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:26:20.748 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.757+0000 7f8d6aa2b640 1 -- 172.21.15.17:0/3062988222 --> v1:172.21.15.17:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d641368c0 con 0x7f8d640a9750 2024-03-29T11:26:20.748 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.757+0000 7f8d6aa2b640 10 client.4532.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:20.748 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.758+0000 7f8d5a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:20.748 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.758+0000 7f8d5a7fc640 1 -- 172.21.15.17:0/3062988222 <== mon.2 v1:172.21.15.17:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f8d5c003de0 con 0x7f8d640a9750 2024-03-29T11:26:20.748 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.758+0000 7f8d5a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:20.748 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.758+0000 7f8d5a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:20.748 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.758+0000 7f8d5a7fc640 10 monclient: dump: 2024-03-29T11:26:20.749 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:26:20.749 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:26:20.749 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:20.749 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:20.749 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:20.749 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:20.749 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:26:20.749 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:26:20.749 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:26:20.749 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:20.749 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.758+0000 7f8d6aa2b640 10 monclient: _renew_subs 2024-03-29T11:26:20.749 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.758+0000 7f8d6aa2b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:26:20.750 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.758+0000 7f8d6aa2b640 1 -- 172.21.15.17:0/3062988222 --> v1:172.21.15.17:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d641327e0 con 0x7f8d640a9750 2024-03-29T11:26:20.750 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.758+0000 7f8d6aa2b640 1 librados: init done 2024-03-29T11:26:20.750 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.758+0000 7f8d5b7fe640 4 set_mon_vals no callback set 2024-03-29T11:26:20.751 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.762+0000 7f8d5a7fc640 1 -- 172.21.15.17:0/3062988222 <== mon.2 v1:172.21.15.17:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f8d5c003f40 con 0x7f8d640a9750 2024-03-29T11:26:20.751 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.762+0000 7f8d6aa2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:20.751 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.762+0000 7f8d6aa2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:20.751 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.763+0000 7f8d6aa2b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:26:20.751 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.763+0000 7f8d6aa2b640 1 -- 172.21.15.17:0/3062988222 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d4c005080 con 0x7f8d640a9750 2024-03-29T11:26:20.751 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.763+0000 7f8d69228640 10 monclient: get_auth_request con 0x7f8d48076630 auth_method 0 2024-03-29T11:26:20.752 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.763+0000 7f8d5a7fc640 1 -- 172.21.15.17:0/3062988222 <== mon.2 v1:172.21.15.17:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f8d5c084c40 con 0x7f8d640a9750 2024-03-29T11:26:20.752 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.763+0000 7f8d5a7fc640 10 client.4532.objecter ms_dispatch 0x7f8d64000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:20.752 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.763+0000 7f8d5a7fc640 3 client.4532.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:20.752 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.763+0000 7f8d5a7fc640 3 client.4532.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:20.752 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.763+0000 7f8d5a7fc640 20 client.4532.objecter dump_active .. 0 homeless 2024-03-29T11:26:20.752 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.763+0000 7f8d5a7fc640 10 client.4532.objecter ms_handle_connect 0x7f8d48076630 2024-03-29T11:26:20.760 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.774+0000 7f8d5a7fc640 1 -- 172.21.15.17:0/3062988222 <== mon.2 v1:172.21.15.17:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8d5c050350 con 0x7f8d640a9750 2024-03-29T11:26:20.760 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.774+0000 7f8d5a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:20.761 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:20.774+0000 7f8d5a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:21.090 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:21.102+0000 7f8d6aa2b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:21.090 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:21.102+0000 7f8d6aa2b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:21.091 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:21.102+0000 7f8d6aa2b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:26:21.091 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:21.102+0000 7f8d6aa2b640 1 -- 172.21.15.17:0/3062988222 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d4c005330 con 0x7f8d640a9750 2024-03-29T11:26:21.093 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:21.107+0000 7f8d5a7fc640 1 -- 172.21.15.17:0/3062988222 <== mon.2 v1:172.21.15.17:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f8d5c052450 con 0x7f8d640a9750 2024-03-29T11:26:21.093 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:21.107+0000 7f8d5a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:21.093 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:21.107+0000 7f8d5a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:21.093 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:26:21.093 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:21.096 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:21.110+0000 7f8d6aa2b640 10 librados: watch_flush enter 2024-03-29T11:26:21.097 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:21.110+0000 7f8d6aa2b640 10 librados: watch_flush exit 2024-03-29T11:26:21.097 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:21.110+0000 7f8d6aa2b640 20 client.4532.objecter shutdown clearing up homeless session... 2024-03-29T11:26:21.097 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:21.110+0000 7f8d6aa2b640 10 client.4532.objecter successfully canceled tick 2024-03-29T11:26:21.097 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:21.111+0000 7f8d6aa2b640 1 -- 172.21.15.17:0/3062988222 >> v1:172.21.15.17:6816/3236185052 conn(0x7f8d48076630 legacy=0x7f8d48078af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:21.097 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:21.111+0000 7f8d6aa2b640 10 monclient: shutdown 2024-03-29T11:26:21.097 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:21.111+0000 7f8d6aa2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:21.097 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:21.111+0000 7f8d6aa2b640 1 -- 172.21.15.17:0/3062988222 >> v1:172.21.15.17:6790/0 conn(0x7f8d640a9750 legacy=0x7f8d64137330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:21.099 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:21.113+0000 7f8d6aa2b640 1 -- 172.21.15.17:0/3062988222 shutdown_connections 2024-03-29T11:26:21.099 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:21.113+0000 7f8d6aa2b640 1 -- 172.21.15.17:0/3062988222 >> 172.21.15.17:0/3062988222 conn(0x7f8d640673e0 msgr2=0x7f8d6413e8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:21.102 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:21.114+0000 7f8d6aa2b640 1 -- 172.21.15.17:0/3062988222 shutdown_connections 2024-03-29T11:26:21.102 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:21.114+0000 7f8d6aa2b640 1 -- 172.21.15.17:0/3062988222 wait complete. 2024-03-29T11:26:21.102 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:21.114+0000 7f8d6aa2b640 1 librados: shutdown 2024-03-29T11:26:21.116 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:27.923 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:28.087 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.100+0000 7f703905c640 10 monclient: get_monmap_and_config 2024-03-29T11:26:28.088 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.100+0000 7f703905c640 10 monclient: build_initial_monmap 2024-03-29T11:26:28.088 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.100+0000 7f703905c640 10 monclient: monmap: 2024-03-29T11:26:28.088 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:26:28.088 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:26:28.088 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:26:28.101267+0000 2024-03-29T11:26:28.088 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:26:28.101267+0000 2024-03-29T11:26:28.088 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:28.088 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:28.088 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:26:28.088 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:26:28.088 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:26:28.089 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:28.089 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.101+0000 7f703905c640 1 Processor -- start 2024-03-29T11:26:28.089 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.101+0000 7f703905c640 1 -- start start 2024-03-29T11:26:28.089 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.101+0000 7f703905c640 10 monclient: init 2024-03-29T11:26:28.090 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.102+0000 7f703905c640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:28.090 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.102+0000 7f703905c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:26:28.090 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.102+0000 7f703905c640 10 monclient(hunting): picked mon.noname-a con 0x7f70340ae250 addr v1:172.21.15.17:6789/0 2024-03-29T11:26:28.090 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.102+0000 7f703905c640 10 monclient(hunting): picked mon.noname-c con 0x7f703413f970 addr v1:172.21.15.94:6789/0 2024-03-29T11:26:28.090 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.102+0000 7f703905c640 10 monclient(hunting): picked mon.noname-b con 0x7f7034142800 addr v1:172.21.15.17:6790/0 2024-03-29T11:26:28.090 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.102+0000 7f703905c640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70340afe00 con 0x7f70340ae250 2024-03-29T11:26:28.090 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.102+0000 7f703905c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7034142c10 con 0x7f703413f970 2024-03-29T11:26:28.091 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.102+0000 7f703905c640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70340afb20 con 0x7f7034142800 2024-03-29T11:26:28.091 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.102+0000 7f703905c640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:28.091 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.102+0000 7f703905c640 10 monclient(hunting): authenticate will time out at 1294.307513s 2024-03-29T11:26:28.091 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.102+0000 7f703259c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f703413f970 0x7f70341420f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:49146/0 (socket says 172.21.15.17:49146) 2024-03-29T11:26:28.091 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.102+0000 7f703259c640 1 -- 172.21.15.17:0/3224771047 learned_addr learned my addr 172.21.15.17:0/3224771047 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:26:28.091 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.103+0000 7f7031d9b640 1 -- 172.21.15.17:0/3224771047 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1489422172 0 0) 0x7f7034142c10 con 0x7f703413f970 2024-03-29T11:26:28.091 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.103+0000 7f7031d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:26:28.091 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.103+0000 7f7031d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:28.091 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.103+0000 7f7031d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:28.091 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.103+0000 7f7031d9b640 10 monclient(hunting): my global_id is 4516 2024-03-29T11:26:28.091 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.103+0000 7f7031d9b640 1 -- 172.21.15.17:0/3224771047 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f701c003520 con 0x7f703413f970 2024-03-29T11:26:28.095 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.107+0000 7f7031d9b640 1 -- 172.21.15.17:0/3224771047 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 959648205 0 0) 0x7f701c003520 con 0x7f703413f970 2024-03-29T11:26:28.095 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.107+0000 7f7031d9b640 1 -- 172.21.15.17:0/3224771047 >> v1:172.21.15.17:6790/0 conn(0x7f7034142800 legacy=0x7f703414da10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:28.095 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.107+0000 7f7031d9b640 1 -- 172.21.15.17:0/3224771047 >> v1:172.21.15.17:6789/0 conn(0x7f70340ae250 legacy=0x7f703413f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:26:28.095 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.108+0000 7f7031d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:26:28.095 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.108+0000 7f7031d9b640 1 monclient: found mon.noname-c 2024-03-29T11:26:28.095 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.108+0000 7f7031d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:28.095 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.108+0000 7f7031d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6789/0 2024-03-29T11:26:28.095 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.108+0000 7f7031d9b640 1 -- 172.21.15.17:0/3224771047 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70340aab80 con 0x7f703413f970 2024-03-29T11:26:28.095 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.108+0000 7f7031d9b640 10 monclient: _finish_auth 0 2024-03-29T11:26:28.095 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.108+0000 7f7031d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:26:28.096 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.108+0000 7f7031d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:28.096 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.108+0000 7f7031d9b640 1 -- 172.21.15.17:0/3224771047 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f7028001690 con 0x7f703413f970 2024-03-29T11:26:28.096 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.108+0000 7f7031d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:28.096 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.108+0000 7f7031d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:28.096 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.108+0000 7f7031d9b640 10 monclient: dump: 2024-03-29T11:26:28.096 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:26:28.096 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:26:28.096 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:28.096 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:28.096 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:28.096 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:28.096 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:26:28.096 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:26:28.096 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:26:28.097 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:28.097 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.108+0000 7f703905c640 5 monclient: authenticate success, global_id 4516 2024-03-29T11:26:28.097 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.108+0000 7f703905c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:28.097 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.108+0000 7f7031d9b640 1 -- 172.21.15.17:0/3224771047 <== mon.1 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7028001920 con 0x7f703413f970 2024-03-29T11:26:28.097 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.108+0000 7f7031d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:28.097 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.108+0000 7f703905c640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:28.097 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.108+0000 7f703905c640 4 set_mon_vals no callback set 2024-03-29T11:26:28.097 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.108+0000 7f7031d9b640 1 -- 172.21.15.17:0/3224771047 <== mon.1 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f7028001cb0 con 0x7f703413f970 2024-03-29T11:26:28.097 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.108+0000 7f7031d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:28.097 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.108+0000 7f7031d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:28.097 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.108+0000 7f7031d9b640 10 monclient: dump: 2024-03-29T11:26:28.097 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:26:28.097 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:26:28.097 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:28.097 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:28.098 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:28.098 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:28.098 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:26:28.098 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:26:28.098 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:26:28.098 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:28.099 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.109+0000 7f703905c640 10 monclient: shutdown 2024-03-29T11:26:28.099 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.109+0000 7f703905c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:28.099 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.109+0000 7f703905c640 1 -- 172.21.15.17:0/3224771047 >> v1:172.21.15.94:6789/0 conn(0x7f703413f970 legacy=0x7f70341420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:28.099 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.110+0000 7f703905c640 1 -- 172.21.15.17:0/3224771047 shutdown_connections 2024-03-29T11:26:28.099 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.110+0000 7f703905c640 1 -- 172.21.15.17:0/3224771047 >> 172.21.15.17:0/3224771047 conn(0x7f70340b35e0 msgr2=0x7f70340b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:28.099 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.110+0000 7f703905c640 1 -- 172.21.15.17:0/3224771047 shutdown_connections 2024-03-29T11:26:28.099 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.111+0000 7f703905c640 1 -- 172.21.15.17:0/3224771047 wait complete. 2024-03-29T11:26:28.099 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.112+0000 7f703905c640 10 monclient: build_initial_monmap 2024-03-29T11:26:28.100 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.112+0000 7f703905c640 10 monclient: monmap: 2024-03-29T11:26:28.100 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:26:28.100 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:28.100 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:26:28.100 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:26:28.100 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:28.100 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:28.100 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:26:28.100 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:26:28.100 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:26:28.100 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:28.100 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.112+0000 7f703905c640 1 librados: starting msgr at 2024-03-29T11:26:28.100 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.112+0000 7f703905c640 1 librados: starting objecter 2024-03-29T11:26:28.100 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.113+0000 7f703905c640 1 Processor -- start 2024-03-29T11:26:28.101 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.113+0000 7f703905c640 1 -- start start 2024-03-29T11:26:28.101 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.113+0000 7f703905c640 1 librados: setting wanted keys 2024-03-29T11:26:28.101 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.113+0000 7f703905c640 1 librados: calling monclient init 2024-03-29T11:26:28.101 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.113+0000 7f703905c640 10 monclient: init 2024-03-29T11:26:28.101 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.114+0000 7f703905c640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:28.101 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.114+0000 7f703905c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:26:28.101 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.114+0000 7f703905c640 10 monclient(hunting): picked mon.noname-b con 0x7f70340ae250 addr v1:172.21.15.94:6789/0 2024-03-29T11:26:28.101 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.114+0000 7f703905c640 10 monclient(hunting): picked mon.noname-c con 0x7f703413f970 addr v1:172.21.15.17:6790/0 2024-03-29T11:26:28.102 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.114+0000 7f703905c640 10 monclient(hunting): picked mon.noname-a con 0x7f7034142800 addr v1:172.21.15.17:6789/0 2024-03-29T11:26:28.102 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.114+0000 7f703905c640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7034144e40 con 0x7f7034142800 2024-03-29T11:26:28.102 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.114+0000 7f703905c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7034143470 con 0x7f70340ae250 2024-03-29T11:26:28.102 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.114+0000 7f703905c640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7034145360 con 0x7f703413f970 2024-03-29T11:26:28.102 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.114+0000 7f703905c640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:28.102 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.114+0000 7f703905c640 10 monclient(hunting): authenticate will time out at 1294.319145s 2024-03-29T11:26:28.103 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.117+0000 7f7032d9d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f70340ae250 0x7f7034144650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:50168/0 (socket says 172.21.15.17:50168) 2024-03-29T11:26:28.103 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.117+0000 7f7032d9d640 1 -- 172.21.15.17:0/3520949359 learned_addr learned my addr 172.21.15.17:0/3520949359 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:26:28.103 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.118+0000 7f701b7fe640 10 client.?.objecter ms_handle_connect 0x7f70340ae250 2024-03-29T11:26:28.104 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.118+0000 7f701b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:28.107 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.120+0000 7f701b7fe640 1 -- 172.21.15.17:0/3520949359 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1240014434 0 0) 0x7f7034143470 con 0x7f70340ae250 2024-03-29T11:26:28.107 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.120+0000 7f701b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:28.107 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.120+0000 7f701b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:28.107 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.120+0000 7f701b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:28.107 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.120+0000 7f701b7fe640 10 monclient(hunting): my global_id is 4519 2024-03-29T11:26:28.107 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.120+0000 7f701b7fe640 1 -- 172.21.15.17:0/3520949359 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7020004eb0 con 0x7f70340ae250 2024-03-29T11:26:28.107 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.120+0000 7f701b7fe640 10 client.?.objecter ms_handle_connect 0x7f703413f970 2024-03-29T11:26:28.107 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.120+0000 7f701b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:28.109 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.121+0000 7f701b7fe640 1 -- 172.21.15.17:0/3520949359 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1835743259 0 0) 0x7f7034145360 con 0x7f703413f970 2024-03-29T11:26:28.109 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.121+0000 7f701b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:28.109 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.121+0000 7f701b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:28.109 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.121+0000 7f701b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:28.109 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.121+0000 7f701b7fe640 10 monclient(hunting): my global_id is 4535 2024-03-29T11:26:28.109 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.121+0000 7f701b7fe640 1 -- 172.21.15.17:0/3520949359 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70200051d0 con 0x7f703413f970 2024-03-29T11:26:28.109 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.122+0000 7f701b7fe640 1 -- 172.21.15.17:0/3520949359 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2670668029 0 0) 0x7f7020004eb0 con 0x7f70340ae250 2024-03-29T11:26:28.109 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.122+0000 7f701b7fe640 1 -- 172.21.15.17:0/3520949359 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7034145360 con 0x7f70340ae250 2024-03-29T11:26:28.109 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.122+0000 7f701b7fe640 1 -- 172.21.15.17:0/3520949359 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f7024004040 con 0x7f70340ae250 2024-03-29T11:26:28.109 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.122+0000 7f701b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:28.109 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.122+0000 7f701b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:28.109 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.122+0000 7f701b7fe640 10 monclient(hunting): dump: 2024-03-29T11:26:28.109 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:26:28.110 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:26:28.110 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:28.110 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:28.110 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:28.110 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:28.110 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:26:28.110 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:26:28.110 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:26:28.110 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:28.110 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.122+0000 7f701b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:28.110 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.122+0000 7f701b7fe640 1 -- 172.21.15.17:0/3520949359 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1897230745 0 0) 0x7f70200051d0 con 0x7f703413f970 2024-03-29T11:26:28.110 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.122+0000 7f701b7fe640 1 -- 172.21.15.17:0/3520949359 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7020004eb0 con 0x7f703413f970 2024-03-29T11:26:28.110 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.122+0000 7f701b7fe640 1 -- 172.21.15.17:0/3520949359 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f7028001550 con 0x7f703413f970 2024-03-29T11:26:28.111 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.122+0000 7f701b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:28.111 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.122+0000 7f701b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:28.111 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.122+0000 7f701b7fe640 10 monclient(hunting): dump: 2024-03-29T11:26:28.111 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:26:28.111 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:26:28.111 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:28.111 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:28.111 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:28.111 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:28.111 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:26:28.111 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:26:28.111 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:26:28.111 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:28.113 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.123+0000 7f701b7fe640 10 client.?.objecter ms_handle_connect 0x7f7034142800 2024-03-29T11:26:28.113 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.123+0000 7f701b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:28.113 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.123+0000 7f701b7fe640 1 -- 172.21.15.17:0/3520949359 <== mon.2 v1:172.21.15.17:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2620661731 0 0) 0x7f7020004eb0 con 0x7f703413f970 2024-03-29T11:26:28.113 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.123+0000 7f701b7fe640 1 -- 172.21.15.17:0/3520949359 >> v1:172.21.15.94:6789/0 conn(0x7f70340ae250 legacy=0x7f7034144650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:28.113 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.123+0000 7f701b7fe640 1 -- 172.21.15.17:0/3520949359 >> v1:172.21.15.17:6789/0 conn(0x7f7034142800 legacy=0x7f70340ab800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:28.113 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.123+0000 7f701b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:26:28.114 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.123+0000 7f701b7fe640 1 monclient: found mon.c 2024-03-29T11:26:28.114 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.123+0000 7f701b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:28.114 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.123+0000 7f701b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:26:28.114 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.123+0000 7f701b7fe640 1 -- 172.21.15.17:0/3520949359 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70341454f0 con 0x7f703413f970 2024-03-29T11:26:28.114 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.123+0000 7f701b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:26:28.114 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.123+0000 7f701b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:26:28.114 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.123+0000 7f701b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:28.114 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.124+0000 7f701b7fe640 1 -- 172.21.15.17:0/3520949359 <== mon.2 v1:172.21.15.17:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7028001730 con 0x7f703413f970 2024-03-29T11:26:28.114 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.124+0000 7f701b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:28.114 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.124+0000 7f701b7fe640 1 -- 172.21.15.17:0/3520949359 <== mon.2 v1:172.21.15.17:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f7028004d90 con 0x7f703413f970 2024-03-29T11:26:28.115 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.124+0000 7f701b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:28.115 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.124+0000 7f701b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:28.115 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.124+0000 7f701b7fe640 10 monclient: dump: 2024-03-29T11:26:28.115 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:26:28.115 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:26:28.115 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:28.115 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:28.115 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:28.115 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:28.115 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:26:28.115 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:26:28.115 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:26:28.115 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:28.116 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.124+0000 7f703159a640 4 set_mon_vals no callback set 2024-03-29T11:26:28.116 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.125+0000 7f703905c640 5 monclient: authenticate success, global_id 4535 2024-03-29T11:26:28.116 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.125+0000 7f703905c640 10 monclient: _renew_subs 2024-03-29T11:26:28.116 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.125+0000 7f703905c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:26:28.116 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.125+0000 7f703905c640 1 -- 172.21.15.17:0/3520949359 --> v1:172.21.15.17:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7034138b40 con 0x7f703413f970 2024-03-29T11:26:28.116 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.125+0000 7f703905c640 10 client.4535.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:28.117 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.125+0000 7f703905c640 10 monclient: _renew_subs 2024-03-29T11:26:28.117 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.125+0000 7f703905c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:26:28.117 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.125+0000 7f703905c640 1 -- 172.21.15.17:0/3520949359 --> v1:172.21.15.17:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f70340ac0f0 con 0x7f703413f970 2024-03-29T11:26:28.117 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.125+0000 7f703905c640 1 librados: init done 2024-03-29T11:26:28.117 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.127+0000 7f701b7fe640 1 -- 172.21.15.17:0/3520949359 <== mon.2 v1:172.21.15.17:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f702801ad60 con 0x7f703413f970 2024-03-29T11:26:28.118 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.128+0000 7f701b7fe640 1 -- 172.21.15.17:0/3520949359 <== mon.2 v1:172.21.15.17:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f7028084b40 con 0x7f703413f970 2024-03-29T11:26:28.118 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.128+0000 7f703905c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:28.118 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.128+0000 7f701b7fe640 10 client.4535.objecter ms_dispatch 0x7f7034000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:28.118 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.128+0000 7f701b7fe640 3 client.4535.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:28.118 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.128+0000 7f701b7fe640 3 client.4535.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:28.118 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.128+0000 7f703905c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:28.118 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.128+0000 7f703905c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:26:28.118 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.128+0000 7f703905c640 1 -- 172.21.15.17:0/3520949359 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7034138cd0 con 0x7f703413f970 2024-03-29T11:26:28.118 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.128+0000 7f701b7fe640 20 client.4535.objecter dump_active .. 0 homeless 2024-03-29T11:26:28.118 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.129+0000 7f7032d9d640 10 monclient: get_auth_request con 0x7f702006e0b0 auth_method 0 2024-03-29T11:26:28.118 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.129+0000 7f701b7fe640 10 client.4535.objecter ms_handle_connect 0x7f702006e0b0 2024-03-29T11:26:28.126 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.140+0000 7f701b7fe640 1 -- 172.21.15.17:0/3520949359 <== mon.2 v1:172.21.15.17:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7028050420 con 0x7f703413f970 2024-03-29T11:26:28.127 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.140+0000 7f701b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:28.127 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.140+0000 7f701b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:28.488 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.501+0000 7f703905c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:28.488 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.501+0000 7f703905c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:28.488 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.501+0000 7f703905c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:26:28.488 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.501+0000 7f703905c640 1 -- 172.21.15.17:0/3520949359 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7034144fd0 con 0x7f703413f970 2024-03-29T11:26:28.489 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:26:28.489 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:28.490 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.502+0000 7f701b7fe640 1 -- 172.21.15.17:0/3520949359 <== mon.2 v1:172.21.15.17:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f7028052350 con 0x7f703413f970 2024-03-29T11:26:28.490 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.503+0000 7f701b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:28.490 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.503+0000 7f701b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:28.493 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.506+0000 7f703905c640 10 librados: watch_flush enter 2024-03-29T11:26:28.493 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.507+0000 7f703905c640 10 librados: watch_flush exit 2024-03-29T11:26:28.493 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.507+0000 7f703905c640 20 client.4535.objecter shutdown clearing up homeless session... 2024-03-29T11:26:28.493 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.507+0000 7f703905c640 10 client.4535.objecter successfully canceled tick 2024-03-29T11:26:28.493 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.507+0000 7f703905c640 1 -- 172.21.15.17:0/3520949359 >> v1:172.21.15.17:6816/3236185052 conn(0x7f702006e0b0 legacy=0x7f7020070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:28.494 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.507+0000 7f703905c640 10 monclient: shutdown 2024-03-29T11:26:28.494 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.507+0000 7f703905c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:28.494 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.507+0000 7f703905c640 1 -- 172.21.15.17:0/3520949359 >> v1:172.21.15.17:6790/0 conn(0x7f703413f970 legacy=0x7f7034138430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:28.494 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.507+0000 7f703905c640 1 -- 172.21.15.17:0/3520949359 shutdown_connections 2024-03-29T11:26:28.494 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.507+0000 7f703905c640 1 -- 172.21.15.17:0/3520949359 >> 172.21.15.17:0/3520949359 conn(0x7f70340b35e0 msgr2=0x7f70340b2770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:28.494 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.508+0000 7f703905c640 1 -- 172.21.15.17:0/3520949359 shutdown_connections 2024-03-29T11:26:28.494 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.508+0000 7f703905c640 1 -- 172.21.15.17:0/3520949359 wait complete. 2024-03-29T11:26:28.498 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:28.512+0000 7f703905c640 1 librados: shutdown 2024-03-29T11:26:28.518 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:35.325 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:35.482 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.495+0000 7f2326bf4640 10 monclient: get_monmap_and_config 2024-03-29T11:26:35.482 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.495+0000 7f2326bf4640 10 monclient: build_initial_monmap 2024-03-29T11:26:35.482 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.495+0000 7f2326bf4640 10 monclient: monmap: 2024-03-29T11:26:35.482 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:26:35.482 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:26:35.482 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:26:35.496795+0000 2024-03-29T11:26:35.483 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:26:35.496795+0000 2024-03-29T11:26:35.483 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:35.483 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:35.483 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:26:35.483 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:26:35.483 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:26:35.483 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:35.483 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.496+0000 7f2326bf4640 1 Processor -- start 2024-03-29T11:26:35.484 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.497+0000 7f2326bf4640 1 -- start start 2024-03-29T11:26:35.484 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.497+0000 7f2326bf4640 10 monclient: init 2024-03-29T11:26:35.484 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.498+0000 7f2326bf4640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:35.485 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.498+0000 7f2326bf4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:26:35.485 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.498+0000 7f2326bf4640 10 monclient(hunting): picked mon.noname-b con 0x7f23200b0770 addr v1:172.21.15.17:6790/0 2024-03-29T11:26:35.485 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.498+0000 7f2326bf4640 10 monclient(hunting): picked mon.noname-c con 0x7f232014a700 addr v1:172.21.15.94:6789/0 2024-03-29T11:26:35.485 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.498+0000 7f2326bf4640 10 monclient(hunting): picked mon.noname-a con 0x7f23200ac5a0 addr v1:172.21.15.17:6789/0 2024-03-29T11:26:35.485 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.498+0000 7f2326bf4640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23200aba70 con 0x7f23200ac5a0 2024-03-29T11:26:35.485 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.498+0000 7f2326bf4640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23200aecb0 con 0x7f232014a700 2024-03-29T11:26:35.485 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.498+0000 7f2326bf4640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23200af150 con 0x7f23200b0770 2024-03-29T11:26:35.485 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.498+0000 7f2326bf4640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:35.485 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.498+0000 7f2326bf4640 10 monclient(hunting): authenticate will time out at 1301.703126s 2024-03-29T11:26:35.485 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.498+0000 7f2325191640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f23200ac5a0 0x7f2320151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:56054/0 (socket says 172.21.15.17:56054) 2024-03-29T11:26:35.485 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.498+0000 7f2325191640 1 -- 172.21.15.17:0/138634498 learned_addr learned my addr 172.21.15.17:0/138634498 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:26:35.487 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.499+0000 7f231f7fe640 1 -- 172.21.15.17:0/138634498 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2769844322 0 0) 0x7f23200aba70 con 0x7f23200ac5a0 2024-03-29T11:26:35.487 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.499+0000 7f231f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:35.487 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.499+0000 7f231f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:35.487 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.499+0000 7f231f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:35.487 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.499+0000 7f231f7fe640 10 monclient(hunting): my global_id is 4485 2024-03-29T11:26:35.487 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.499+0000 7f231f7fe640 1 -- 172.21.15.17:0/138634498 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2310003520 con 0x7f23200ac5a0 2024-03-29T11:26:35.487 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.499+0000 7f231f7fe640 1 -- 172.21.15.17:0/138634498 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3801155882 0 0) 0x7f23200af150 con 0x7f23200b0770 2024-03-29T11:26:35.488 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.499+0000 7f231f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:35.488 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.499+0000 7f231f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:35.488 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.499+0000 7f231f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:35.488 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.499+0000 7f231f7fe640 10 monclient(hunting): my global_id is 4538 2024-03-29T11:26:35.488 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.499+0000 7f231f7fe640 1 -- 172.21.15.17:0/138634498 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23200aba70 con 0x7f23200b0770 2024-03-29T11:26:35.488 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.499+0000 7f231f7fe640 1 -- 172.21.15.17:0/138634498 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2672445204 0 0) 0x7f23200aecb0 con 0x7f232014a700 2024-03-29T11:26:35.488 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.499+0000 7f231f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:35.488 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.499+0000 7f231f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:35.488 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.499+0000 7f231f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:35.488 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.499+0000 7f231f7fe640 10 monclient(hunting): my global_id is 4522 2024-03-29T11:26:35.489 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.499+0000 7f231f7fe640 1 -- 172.21.15.17:0/138634498 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23200af150 con 0x7f232014a700 2024-03-29T11:26:35.489 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.500+0000 7f231f7fe640 1 -- 172.21.15.17:0/138634498 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3956631149 0 0) 0x7f23200af150 con 0x7f232014a700 2024-03-29T11:26:35.489 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.500+0000 7f231f7fe640 1 -- 172.21.15.17:0/138634498 >> v1:172.21.15.17:6790/0 conn(0x7f23200b0770 legacy=0x7f2320149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:35.489 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.500+0000 7f231f7fe640 1 -- 172.21.15.17:0/138634498 >> v1:172.21.15.17:6789/0 conn(0x7f23200ac5a0 legacy=0x7f2320151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:35.489 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.500+0000 7f231f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:26:35.489 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.500+0000 7f231f7fe640 1 monclient: found mon.noname-c 2024-03-29T11:26:35.489 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.500+0000 7f231f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:35.489 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.500+0000 7f231f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6789/0 2024-03-29T11:26:35.489 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.500+0000 7f231f7fe640 1 -- 172.21.15.17:0/138634498 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23200ac9d0 con 0x7f232014a700 2024-03-29T11:26:35.489 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.500+0000 7f231f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:26:35.489 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.500+0000 7f231f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:26:35.489 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.500+0000 7f231f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:35.489 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.500+0000 7f2326bf4640 5 monclient: authenticate success, global_id 4522 2024-03-29T11:26:35.490 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.500+0000 7f2326bf4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:35.490 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.500+0000 7f231f7fe640 1 -- 172.21.15.17:0/138634498 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f2314001550 con 0x7f232014a700 2024-03-29T11:26:35.490 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.500+0000 7f231f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:35.490 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.500+0000 7f231f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:35.490 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.500+0000 7f231f7fe640 10 monclient: dump: 2024-03-29T11:26:35.490 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:26:35.490 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:26:35.490 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:35.490 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:35.490 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:35.490 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:35.491 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:26:35.491 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:26:35.491 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:26:35.491 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:35.491 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.500+0000 7f2326bf4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:35.491 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.501+0000 7f231f7fe640 1 -- 172.21.15.17:0/138634498 <== mon.1 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2314001730 con 0x7f232014a700 2024-03-29T11:26:35.491 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.501+0000 7f231f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:35.499 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.501+0000 7f2326bf4640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:35.499 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.501+0000 7f2326bf4640 4 set_mon_vals no callback set 2024-03-29T11:26:35.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.501+0000 7f231f7fe640 1 -- 172.21.15.17:0/138634498 <== mon.1 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f2314001b40 con 0x7f232014a700 2024-03-29T11:26:35.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.501+0000 7f231f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:35.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.501+0000 7f231f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:35.500 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.501+0000 7f231f7fe640 10 monclient: dump: 2024-03-29T11:26:35.500 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:26:35.500 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:26:35.500 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:35.500 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:35.500 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:35.501 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:35.501 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:26:35.501 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:26:35.501 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:26:35.501 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:35.501 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.501+0000 7f2326bf4640 10 monclient: shutdown 2024-03-29T11:26:35.501 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.501+0000 7f2326bf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:35.501 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.501+0000 7f2326bf4640 1 -- 172.21.15.17:0/138634498 >> v1:172.21.15.94:6789/0 conn(0x7f232014a700 legacy=0x7f23200abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:35.501 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.502+0000 7f2326bf4640 1 -- 172.21.15.17:0/138634498 shutdown_connections 2024-03-29T11:26:35.501 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.502+0000 7f2326bf4640 1 -- 172.21.15.17:0/138634498 >> 172.21.15.17:0/138634498 conn(0x7f232006b660 msgr2=0x7f232006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:35.502 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.502+0000 7f2326bf4640 1 -- 172.21.15.17:0/138634498 shutdown_connections 2024-03-29T11:26:35.502 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.502+0000 7f2326bf4640 1 -- 172.21.15.17:0/138634498 wait complete. 2024-03-29T11:26:35.502 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.503+0000 7f2326bf4640 10 monclient: build_initial_monmap 2024-03-29T11:26:35.502 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.503+0000 7f2326bf4640 10 monclient: monmap: 2024-03-29T11:26:35.502 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:26:35.502 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:35.502 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:26:35.503 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:26:35.503 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:35.503 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:35.503 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:26:35.503 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:26:35.503 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:26:35.503 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:35.503 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.503+0000 7f2326bf4640 1 librados: starting msgr at 2024-03-29T11:26:35.503 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.503+0000 7f2326bf4640 1 librados: starting objecter 2024-03-29T11:26:35.503 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.503+0000 7f2326bf4640 1 Processor -- start 2024-03-29T11:26:35.503 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.504+0000 7f2326bf4640 1 -- start start 2024-03-29T11:26:35.503 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.504+0000 7f2326bf4640 1 librados: setting wanted keys 2024-03-29T11:26:35.503 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.504+0000 7f2326bf4640 1 librados: calling monclient init 2024-03-29T11:26:35.504 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.504+0000 7f2326bf4640 10 monclient: init 2024-03-29T11:26:35.504 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.505+0000 7f2326bf4640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:35.504 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.505+0000 7f2326bf4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:26:35.504 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.505+0000 7f2326bf4640 10 monclient(hunting): picked mon.noname-c con 0x7f23200ac5a0 addr v1:172.21.15.17:6790/0 2024-03-29T11:26:35.504 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.505+0000 7f2326bf4640 10 monclient(hunting): picked mon.noname-a con 0x7f23200b0770 addr v1:172.21.15.17:6789/0 2024-03-29T11:26:35.504 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.505+0000 7f2326bf4640 10 monclient(hunting): picked mon.noname-b con 0x7f232014a700 addr v1:172.21.15.94:6789/0 2024-03-29T11:26:35.504 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.505+0000 7f2326bf4640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2320125b20 con 0x7f23200b0770 2024-03-29T11:26:35.504 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.505+0000 7f2326bf4640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2320127460 con 0x7f232014a700 2024-03-29T11:26:35.504 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.505+0000 7f2326bf4640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23201dbc80 con 0x7f23200ac5a0 2024-03-29T11:26:35.504 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.505+0000 7f2326bf4640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:35.504 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.505+0000 7f2326bf4640 10 monclient(hunting): authenticate will time out at 1301.710011s 2024-03-29T11:26:35.505 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.505+0000 7f231ffff640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f23200b0770 0x7f2320144130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:56626/0 (socket says 172.21.15.17:56626) 2024-03-29T11:26:35.505 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.505+0000 7f231ffff640 1 -- 172.21.15.17:0/3663107825 learned_addr learned my addr 172.21.15.17:0/3663107825 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:26:35.505 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.505+0000 7f231d7fa640 10 client.?.objecter ms_handle_connect 0x7f23200b0770 2024-03-29T11:26:35.505 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.505+0000 7f231d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:35.506 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.505+0000 7f231d7fa640 10 client.?.objecter ms_handle_connect 0x7f232014a700 2024-03-29T11:26:35.506 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.505+0000 7f231d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:35.506 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.506+0000 7f231d7fa640 1 -- 172.21.15.17:0/3663107825 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2635154804 0 0) 0x7f2320127460 con 0x7f232014a700 2024-03-29T11:26:35.506 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.506+0000 7f231d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:26:35.506 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.506+0000 7f231d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:35.506 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.506+0000 7f231d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:35.506 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.506+0000 7f231d7fa640 10 monclient(hunting): my global_id is 4525 2024-03-29T11:26:35.506 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.506+0000 7f231d7fa640 1 -- 172.21.15.17:0/3663107825 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2304004eb0 con 0x7f232014a700 2024-03-29T11:26:35.506 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.506+0000 7f231d7fa640 1 -- 172.21.15.17:0/3663107825 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 379973732 0 0) 0x7f2320125b20 con 0x7f23200b0770 2024-03-29T11:26:35.506 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.506+0000 7f231d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:26:35.507 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.506+0000 7f231d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:35.507 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.506+0000 7f231d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:35.507 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.506+0000 7f231d7fa640 10 monclient(hunting): my global_id is 4488 2024-03-29T11:26:35.507 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.506+0000 7f231d7fa640 1 -- 172.21.15.17:0/3663107825 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2320127460 con 0x7f23200b0770 2024-03-29T11:26:35.507 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.506+0000 7f231d7fa640 1 -- 172.21.15.17:0/3663107825 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3886461265 0 0) 0x7f2304004eb0 con 0x7f232014a700 2024-03-29T11:26:35.507 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.507+0000 7f231d7fa640 1 -- 172.21.15.17:0/3663107825 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2320125b20 con 0x7f232014a700 2024-03-29T11:26:35.507 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.507+0000 7f231d7fa640 1 -- 172.21.15.17:0/3663107825 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f2318001cd0 con 0x7f232014a700 2024-03-29T11:26:35.507 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.507+0000 7f231d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:35.507 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.507+0000 7f231d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:35.507 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.507+0000 7f231d7fa640 10 monclient(hunting): dump: 2024-03-29T11:26:35.507 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:26:35.507 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:26:35.507 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:35.507 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:35.507 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:35.508 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:35.508 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:26:35.508 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:26:35.508 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:26:35.508 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:35.508 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.507+0000 7f231d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:35.508 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.507+0000 7f231d7fa640 10 client.?.objecter ms_handle_connect 0x7f23200ac5a0 2024-03-29T11:26:35.508 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.507+0000 7f231d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:35.508 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.507+0000 7f231d7fa640 1 -- 172.21.15.17:0/3663107825 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3482859556 0 0) 0x7f2320127460 con 0x7f23200b0770 2024-03-29T11:26:35.509 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.507+0000 7f231d7fa640 1 -- 172.21.15.17:0/3663107825 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2304004eb0 con 0x7f23200b0770 2024-03-29T11:26:35.509 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.507+0000 7f231d7fa640 1 -- 172.21.15.17:0/3663107825 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f2314001430 con 0x7f23200b0770 2024-03-29T11:26:35.509 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.507+0000 7f231d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:35.509 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.507+0000 7f231d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:35.509 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.507+0000 7f231d7fa640 10 monclient(hunting): dump: 2024-03-29T11:26:35.509 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:26:35.509 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:26:35.509 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:35.509 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:35.510 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:35.510 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:35.510 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:26:35.510 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:26:35.510 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:26:35.510 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:35.510 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.507+0000 7f231d7fa640 1 -- 172.21.15.17:0/3663107825 <== mon.1 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1650968025 0 0) 0x7f2320125b20 con 0x7f232014a700 2024-03-29T11:26:35.510 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.507+0000 7f231d7fa640 1 -- 172.21.15.17:0/3663107825 >> v1:172.21.15.17:6790/0 conn(0x7f23200ac5a0 legacy=0x7f23201db570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:35.510 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.507+0000 7f231d7fa640 1 -- 172.21.15.17:0/3663107825 >> v1:172.21.15.17:6789/0 conn(0x7f23200b0770 legacy=0x7f2320144130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:35.511 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.507+0000 7f231d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:26:35.511 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.507+0000 7f231d7fa640 1 monclient: found mon.b 2024-03-29T11:26:35.511 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.507+0000 7f231d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:35.511 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.507+0000 7f231d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:26:35.511 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.507+0000 7f231d7fa640 1 -- 172.21.15.17:0/3663107825 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23201d90b0 con 0x7f232014a700 2024-03-29T11:26:35.511 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.507+0000 7f231d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:26:35.511 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.507+0000 7f231d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:26:35.511 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.507+0000 7f231d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:35.511 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.508+0000 7f231d7fa640 1 -- 172.21.15.17:0/3663107825 <== mon.1 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2318004b60 con 0x7f232014a700 2024-03-29T11:26:35.512 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.508+0000 7f231d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:35.512 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.508+0000 7f231d7fa640 1 -- 172.21.15.17:0/3663107825 <== mon.1 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f2318004e70 con 0x7f232014a700 2024-03-29T11:26:35.512 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.508+0000 7f231d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:35.512 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.508+0000 7f231d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:35.512 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.508+0000 7f231d7fa640 10 monclient: dump: 2024-03-29T11:26:35.512 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:26:35.512 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:26:35.512 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:35.512 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:35.512 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:35.512 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:35.512 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:26:35.512 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:26:35.512 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:26:35.512 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:35.513 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.508+0000 7f231effd640 4 set_mon_vals no callback set 2024-03-29T11:26:35.513 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.508+0000 7f2326bf4640 5 monclient: authenticate success, global_id 4525 2024-03-29T11:26:35.513 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.508+0000 7f2326bf4640 10 monclient: _renew_subs 2024-03-29T11:26:35.513 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.508+0000 7f2326bf4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:26:35.513 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.508+0000 7f2326bf4640 1 -- 172.21.15.17:0/3663107825 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23201256b0 con 0x7f232014a700 2024-03-29T11:26:35.513 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.508+0000 7f2326bf4640 10 client.4525.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:35.513 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.508+0000 7f2326bf4640 10 monclient: _renew_subs 2024-03-29T11:26:35.513 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.508+0000 7f2326bf4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:26:35.513 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.508+0000 7f2326bf4640 1 -- 172.21.15.17:0/3663107825 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f23201253c0 con 0x7f232014a700 2024-03-29T11:26:35.513 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.508+0000 7f2326bf4640 1 librados: init done 2024-03-29T11:26:35.513 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.510+0000 7f2326bf4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:35.513 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.510+0000 7f2326bf4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:35.513 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.510+0000 7f2326bf4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:26:35.513 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.510+0000 7f2326bf4640 1 -- 172.21.15.17:0/3663107825 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23200ab560 con 0x7f232014a700 2024-03-29T11:26:35.514 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.511+0000 7f231d7fa640 1 -- 172.21.15.17:0/3663107825 <== mon.1 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f231801ae40 con 0x7f232014a700 2024-03-29T11:26:35.514 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.511+0000 7f2324990640 10 monclient: get_auth_request con 0x7f230406de90 auth_method 0 2024-03-29T11:26:35.514 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.511+0000 7f231d7fa640 1 -- 172.21.15.17:0/3663107825 <== mon.1 v1:172.21.15.94:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f2318084d50 con 0x7f232014a700 2024-03-29T11:26:35.514 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.511+0000 7f231d7fa640 10 client.4525.objecter ms_dispatch 0x7f2320000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:35.514 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.511+0000 7f231d7fa640 3 client.4525.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:35.514 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.511+0000 7f231d7fa640 3 client.4525.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:35.514 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.511+0000 7f231d7fa640 20 client.4525.objecter dump_active .. 0 homeless 2024-03-29T11:26:35.514 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.512+0000 7f231d7fa640 10 client.4525.objecter ms_handle_connect 0x7f230406de90 2024-03-29T11:26:35.515 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.518+0000 7f231d7fa640 1 -- 172.21.15.17:0/3663107825 <== mon.1 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2318050630 con 0x7f232014a700 2024-03-29T11:26:35.515 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.518+0000 7f231d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:35.515 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.518+0000 7f231d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:35.888 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.901+0000 7f2326bf4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:35.888 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.901+0000 7f2326bf4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:35.888 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.901+0000 7f2326bf4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:26:35.888 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.901+0000 7f2326bf4640 1 -- 172.21.15.17:0/3663107825 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2320075f10 con 0x7f232014a700 2024-03-29T11:26:35.889 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.902+0000 7f231d7fa640 1 -- 172.21.15.17:0/3663107825 <== mon.1 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f2318052560 con 0x7f232014a700 2024-03-29T11:26:35.889 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.902+0000 7f231d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:35.889 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.902+0000 7f231d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:35.889 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:26:35.889 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:35.892 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.905+0000 7f2326bf4640 10 librados: watch_flush enter 2024-03-29T11:26:35.892 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.906+0000 7f2326bf4640 10 librados: watch_flush exit 2024-03-29T11:26:35.892 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.906+0000 7f2326bf4640 20 client.4525.objecter shutdown clearing up homeless session... 2024-03-29T11:26:35.892 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.906+0000 7f2326bf4640 10 client.4525.objecter successfully canceled tick 2024-03-29T11:26:35.893 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.906+0000 7f2326bf4640 1 -- 172.21.15.17:0/3663107825 >> v1:172.21.15.17:6816/3236185052 conn(0x7f230406de90 legacy=0x7f2304070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:35.893 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.906+0000 7f2326bf4640 10 monclient: shutdown 2024-03-29T11:26:35.893 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.906+0000 7f2326bf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:35.893 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.906+0000 7f2326bf4640 1 -- 172.21.15.17:0/3663107825 >> v1:172.21.15.94:6789/0 conn(0x7f232014a700 legacy=0x7f23201e6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:35.893 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.906+0000 7f2326bf4640 1 -- 172.21.15.17:0/3663107825 shutdown_connections 2024-03-29T11:26:35.893 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.906+0000 7f2326bf4640 1 -- 172.21.15.17:0/3663107825 >> 172.21.15.17:0/3663107825 conn(0x7f232006b660 msgr2=0x7f2320149530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:35.893 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.906+0000 7f2326bf4640 1 -- 172.21.15.17:0/3663107825 shutdown_connections 2024-03-29T11:26:35.893 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.906+0000 7f2326bf4640 1 -- 172.21.15.17:0/3663107825 wait complete. 2024-03-29T11:26:35.893 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:35.907+0000 7f2326bf4640 1 librados: shutdown 2024-03-29T11:26:35.907 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:42.714 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:42.856 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.869+0000 7fc58ed6b640 10 monclient: get_monmap_and_config 2024-03-29T11:26:42.856 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.869+0000 7fc58ed6b640 10 monclient: build_initial_monmap 2024-03-29T11:26:42.856 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.869+0000 7fc58ed6b640 10 monclient: monmap: 2024-03-29T11:26:42.856 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:26:42.856 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:26:42.856 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:26:42.870713+0000 2024-03-29T11:26:42.856 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:26:42.870713+0000 2024-03-29T11:26:42.857 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:42.857 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:42.857 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:26:42.857 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:26:42.857 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:26:42.857 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:42.857 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.871+0000 7fc58ed6b640 1 Processor -- start 2024-03-29T11:26:42.857 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.871+0000 7fc58ed6b640 1 -- start start 2024-03-29T11:26:42.858 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.871+0000 7fc58ed6b640 10 monclient: init 2024-03-29T11:26:42.859 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.872+0000 7fc58ed6b640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:42.859 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.872+0000 7fc58ed6b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:26:42.859 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.872+0000 7fc58ed6b640 10 monclient(hunting): picked mon.noname-b con 0x7fc5881422b0 addr v1:172.21.15.17:6790/0 2024-03-29T11:26:42.859 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.872+0000 7fc58ed6b640 10 monclient(hunting): picked mon.noname-c con 0x7fc588142690 addr v1:172.21.15.94:6789/0 2024-03-29T11:26:42.859 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.872+0000 7fc58ed6b640 10 monclient(hunting): picked mon.noname-a con 0x7fc5880b17d0 addr v1:172.21.15.17:6789/0 2024-03-29T11:26:42.859 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.872+0000 7fc58ed6b640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc588142c80 con 0x7fc5880b17d0 2024-03-29T11:26:42.859 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.872+0000 7fc58ed6b640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5880aae40 con 0x7fc588142690 2024-03-29T11:26:42.859 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.872+0000 7fc58ed6b640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc58813fbd0 con 0x7fc5881422b0 2024-03-29T11:26:42.860 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.872+0000 7fc58ed6b640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:42.860 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.872+0000 7fc58ed6b640 10 monclient(hunting): authenticate will time out at 1309.077550s 2024-03-29T11:26:42.860 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.872+0000 7fc58cb07640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7fc5881422b0 0x7fc58813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:57478/0 (socket says 172.21.15.17:57478) 2024-03-29T11:26:42.860 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.872+0000 7fc58cb07640 1 -- 172.21.15.17:0/2794394882 learned_addr learned my addr 172.21.15.17:0/2794394882 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:26:42.864 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.873+0000 7fc5877fe640 1 -- 172.21.15.17:0/2794394882 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3244651861 0 0) 0x7fc58813fbd0 con 0x7fc5881422b0 2024-03-29T11:26:42.865 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.873+0000 7fc5877fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:42.865 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.873+0000 7fc5877fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:42.865 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.873+0000 7fc5877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:42.865 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.873+0000 7fc5877fe640 10 monclient(hunting): my global_id is 4541 2024-03-29T11:26:42.865 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.873+0000 7fc5877fe640 1 -- 172.21.15.17:0/2794394882 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc570003520 con 0x7fc5881422b0 2024-03-29T11:26:42.865 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.873+0000 7fc5877fe640 1 -- 172.21.15.17:0/2794394882 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2479130799 0 0) 0x7fc570003520 con 0x7fc5881422b0 2024-03-29T11:26:42.865 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.873+0000 7fc5877fe640 1 -- 172.21.15.17:0/2794394882 >> v1:172.21.15.94:6789/0 conn(0x7fc588142690 legacy=0x7fc5880b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:26:42.865 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.873+0000 7fc5877fe640 1 -- 172.21.15.17:0/2794394882 >> v1:172.21.15.17:6789/0 conn(0x7fc5880b17d0 legacy=0x7fc5880b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:42.865 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.873+0000 7fc5877fe640 10 monclient: _finish_hunting 0 2024-03-29T11:26:42.866 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.873+0000 7fc5877fe640 1 monclient: found mon.noname-b 2024-03-29T11:26:42.866 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.874+0000 7fc5877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:42.866 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.874+0000 7fc5877fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.17:6790/0 2024-03-29T11:26:42.866 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.874+0000 7fc5877fe640 1 -- 172.21.15.17:0/2794394882 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc588151940 con 0x7fc5881422b0 2024-03-29T11:26:42.866 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.874+0000 7fc5877fe640 10 monclient: _finish_auth 0 2024-03-29T11:26:42.866 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.874+0000 7fc5877fe640 10 monclient: _check_auth_tickets 2024-03-29T11:26:42.866 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.874+0000 7fc5877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:42.866 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.874+0000 7fc5877fe640 1 -- 172.21.15.17:0/2794394882 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fc578001a10 con 0x7fc5881422b0 2024-03-29T11:26:42.867 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.874+0000 7fc5877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:42.867 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.874+0000 7fc5877fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:42.867 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.874+0000 7fc5877fe640 10 monclient: dump: 2024-03-29T11:26:42.867 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:26:42.867 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:26:42.867 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:42.867 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:42.867 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:42.867 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:42.868 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:26:42.868 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:26:42.868 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:26:42.868 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:42.868 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.874+0000 7fc5877fe640 1 -- 172.21.15.17:0/2794394882 <== mon.2 v1:172.21.15.17:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc578001c30 con 0x7fc5881422b0 2024-03-29T11:26:42.868 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.874+0000 7fc5877fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:42.868 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.874+0000 7fc5877fe640 1 -- 172.21.15.17:0/2794394882 <== mon.2 v1:172.21.15.17:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fc578001fc0 con 0x7fc5881422b0 2024-03-29T11:26:42.868 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.874+0000 7fc5877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:42.869 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.874+0000 7fc5877fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:42.869 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.874+0000 7fc5877fe640 10 monclient: dump: 2024-03-29T11:26:42.869 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:26:42.869 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:26:42.869 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:42.869 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:42.869 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:42.869 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:42.869 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:26:42.870 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:26:42.870 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:26:42.870 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:42.870 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.874+0000 7fc58ed6b640 5 monclient: authenticate success, global_id 4541 2024-03-29T11:26:42.870 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.874+0000 7fc58ed6b640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:42.870 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.874+0000 7fc58ed6b640 4 set_mon_vals no callback set 2024-03-29T11:26:42.870 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.875+0000 7fc58ed6b640 10 monclient: shutdown 2024-03-29T11:26:42.870 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.875+0000 7fc58ed6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:42.870 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.875+0000 7fc58ed6b640 1 -- 172.21.15.17:0/2794394882 >> v1:172.21.15.17:6790/0 conn(0x7fc5881422b0 legacy=0x7fc58813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:42.871 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.876+0000 7fc58ed6b640 1 -- 172.21.15.17:0/2794394882 shutdown_connections 2024-03-29T11:26:42.871 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.876+0000 7fc58ed6b640 1 -- 172.21.15.17:0/2794394882 >> 172.21.15.17:0/2794394882 conn(0x7fc588066a00 msgr2=0x7fc588066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:42.871 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.876+0000 7fc58ed6b640 1 -- 172.21.15.17:0/2794394882 shutdown_connections 2024-03-29T11:26:42.871 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.876+0000 7fc58ed6b640 1 -- 172.21.15.17:0/2794394882 wait complete. 2024-03-29T11:26:42.872 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.877+0000 7fc58ed6b640 10 monclient: build_initial_monmap 2024-03-29T11:26:42.872 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.877+0000 7fc58ed6b640 10 monclient: monmap: 2024-03-29T11:26:42.872 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:26:42.872 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:42.872 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:26:42.872 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:26:42.872 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:42.872 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:42.873 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:26:42.873 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:26:42.873 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:26:42.873 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:42.885 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.877+0000 7fc58ed6b640 1 librados: starting msgr at 2024-03-29T11:26:42.885 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.877+0000 7fc58ed6b640 1 librados: starting objecter 2024-03-29T11:26:42.885 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.877+0000 7fc58ed6b640 1 Processor -- start 2024-03-29T11:26:42.885 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.877+0000 7fc58ed6b640 1 -- start start 2024-03-29T11:26:42.885 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.877+0000 7fc58ed6b640 1 librados: setting wanted keys 2024-03-29T11:26:42.885 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.877+0000 7fc58ed6b640 1 librados: calling monclient init 2024-03-29T11:26:42.885 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.877+0000 7fc58ed6b640 10 monclient: init 2024-03-29T11:26:42.886 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.878+0000 7fc58ed6b640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:42.886 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.878+0000 7fc58ed6b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:26:42.886 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.878+0000 7fc58ed6b640 10 monclient(hunting): picked mon.noname-b con 0x7fc5880b17d0 addr v1:172.21.15.94:6789/0 2024-03-29T11:26:42.886 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.878+0000 7fc58ed6b640 10 monclient(hunting): picked mon.noname-a con 0x7fc5881422b0 addr v1:172.21.15.17:6789/0 2024-03-29T11:26:42.886 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.878+0000 7fc58ed6b640 10 monclient(hunting): picked mon.noname-c con 0x7fc588142690 addr v1:172.21.15.17:6790/0 2024-03-29T11:26:42.886 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.878+0000 7fc587fff640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fc5881422b0 0x7fc5880b5780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:59082/0 (socket says 172.21.15.17:59082) 2024-03-29T11:26:42.886 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.879+0000 7fc587fff640 1 -- 172.21.15.17:0/1379868222 learned_addr learned my addr 172.21.15.17:0/1379868222 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:26:42.886 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.879+0000 7fc58ed6b640 1 -- 172.21.15.17:0/1379868222 --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5880b7920 con 0x7fc5881422b0 2024-03-29T11:26:42.886 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.879+0000 7fc58ed6b640 1 -- 172.21.15.17:0/1379868222 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5880b7400 con 0x7fc5880b17d0 2024-03-29T11:26:42.887 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.879+0000 7fc58ed6b640 1 -- 172.21.15.17:0/1379868222 --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5880b7090 con 0x7fc588142690 2024-03-29T11:26:42.887 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.879+0000 7fc58ed6b640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:42.887 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.879+0000 7fc58ed6b640 10 monclient(hunting): authenticate will time out at 1309.084028s 2024-03-29T11:26:42.887 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.879+0000 7fc5857fa640 10 client.?.objecter ms_handle_connect 0x7fc5881422b0 2024-03-29T11:26:42.887 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.879+0000 7fc5857fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:42.887 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.879+0000 7fc5857fa640 10 client.?.objecter ms_handle_connect 0x7fc588142690 2024-03-29T11:26:42.887 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.879+0000 7fc5857fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:42.887 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.879+0000 7fc5857fa640 1 -- 172.21.15.17:0/1379868222 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 580361928 0 0) 0x7fc5880b7920 con 0x7fc5881422b0 2024-03-29T11:26:42.887 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.879+0000 7fc5857fa640 10 monclient(hunting): _negotiate 2024-03-29T11:26:42.887 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.879+0000 7fc5857fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:42.887 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.879+0000 7fc5857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:42.888 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.879+0000 7fc5857fa640 10 monclient(hunting): my global_id is 4494 2024-03-29T11:26:42.888 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.879+0000 7fc5857fa640 1 -- 172.21.15.17:0/1379868222 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc574004eb0 con 0x7fc5881422b0 2024-03-29T11:26:42.888 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.879+0000 7fc5857fa640 10 client.?.objecter ms_handle_connect 0x7fc5880b17d0 2024-03-29T11:26:42.888 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.880+0000 7fc5857fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:42.888 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.880+0000 7fc5857fa640 1 -- 172.21.15.17:0/1379868222 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3061414539 0 0) 0x7fc5880b7400 con 0x7fc5880b17d0 2024-03-29T11:26:42.888 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.880+0000 7fc5857fa640 10 monclient(hunting): _negotiate 2024-03-29T11:26:42.888 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.880+0000 7fc5857fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:42.888 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.880+0000 7fc5857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:42.888 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.880+0000 7fc5857fa640 10 monclient(hunting): my global_id is 4528 2024-03-29T11:26:42.888 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.880+0000 7fc5857fa640 1 -- 172.21.15.17:0/1379868222 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5880b7920 con 0x7fc5880b17d0 2024-03-29T11:26:42.888 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.880+0000 7fc5857fa640 1 -- 172.21.15.17:0/1379868222 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1621595466 0 0) 0x7fc574004eb0 con 0x7fc5881422b0 2024-03-29T11:26:42.888 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.880+0000 7fc5857fa640 1 -- 172.21.15.17:0/1379868222 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5880b7400 con 0x7fc5881422b0 2024-03-29T11:26:42.889 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.880+0000 7fc5857fa640 1 -- 172.21.15.17:0/1379868222 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fc57c0017d0 con 0x7fc5881422b0 2024-03-29T11:26:42.889 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.880+0000 7fc5857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:42.889 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.880+0000 7fc5857fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:42.889 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.880+0000 7fc5857fa640 10 monclient(hunting): dump: 2024-03-29T11:26:42.889 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:26:42.889 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:26:42.889 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:42.889 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:42.889 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:42.889 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:42.889 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:26:42.889 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:26:42.889 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:26:42.889 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:42.889 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.880+0000 7fc5857fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:42.889 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.880+0000 7fc5857fa640 1 -- 172.21.15.17:0/1379868222 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1448665503 0 0) 0x7fc5880b7920 con 0x7fc5880b17d0 2024-03-29T11:26:42.890 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.880+0000 7fc5857fa640 1 -- 172.21.15.17:0/1379868222 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc574004eb0 con 0x7fc5880b17d0 2024-03-29T11:26:42.890 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.880+0000 7fc5857fa640 1 -- 172.21.15.17:0/1379868222 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fc5780018d0 con 0x7fc5880b17d0 2024-03-29T11:26:42.890 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.880+0000 7fc5857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:42.890 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.880+0000 7fc5857fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:42.890 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.880+0000 7fc5857fa640 10 monclient(hunting): dump: 2024-03-29T11:26:42.890 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:26:42.890 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:26:42.890 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:42.890 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:42.890 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:42.890 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:42.890 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:26:42.890 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:26:42.891 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:26:42.891 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:42.891 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.880+0000 7fc5857fa640 1 -- 172.21.15.17:0/1379868222 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3897020680 0 0) 0x7fc5880b7090 con 0x7fc588142690 2024-03-29T11:26:42.891 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.880+0000 7fc5857fa640 10 monclient(hunting): _negotiate 2024-03-29T11:26:42.891 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.880+0000 7fc5857fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:42.891 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.880+0000 7fc5857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:42.891 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.880+0000 7fc5857fa640 10 monclient(hunting): my global_id is 4544 2024-03-29T11:26:42.891 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.880+0000 7fc5857fa640 1 -- 172.21.15.17:0/1379868222 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5880b7920 con 0x7fc588142690 2024-03-29T11:26:42.891 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.880+0000 7fc5857fa640 1 -- 172.21.15.17:0/1379868222 <== mon.0 v1:172.21.15.17:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 726509635 0 0) 0x7fc5880b7400 con 0x7fc5881422b0 2024-03-29T11:26:42.891 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.880+0000 7fc5857fa640 1 -- 172.21.15.17:0/1379868222 >> v1:172.21.15.17:6790/0 conn(0x7fc588142690 legacy=0x7fc5881e2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:42.891 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.880+0000 7fc5857fa640 1 -- 172.21.15.17:0/1379868222 >> v1:172.21.15.94:6789/0 conn(0x7fc5880b17d0 legacy=0x7fc5880b09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:42.891 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.880+0000 7fc5857fa640 10 monclient: _finish_hunting 0 2024-03-29T11:26:42.891 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.880+0000 7fc5857fa640 1 monclient: found mon.a 2024-03-29T11:26:42.892 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.880+0000 7fc5857fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:42.892 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.880+0000 7fc5857fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:26:42.892 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.880+0000 7fc5857fa640 1 -- 172.21.15.17:0/1379868222 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc588150a30 con 0x7fc5881422b0 2024-03-29T11:26:42.892 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.881+0000 7fc5857fa640 10 monclient: _finish_auth 0 2024-03-29T11:26:42.892 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.881+0000 7fc5857fa640 10 monclient: _check_auth_tickets 2024-03-29T11:26:42.892 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.881+0000 7fc5857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:42.892 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.881+0000 7fc58ed6b640 5 monclient: authenticate success, global_id 4494 2024-03-29T11:26:42.892 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.881+0000 7fc58ed6b640 10 monclient: _renew_subs 2024-03-29T11:26:42.892 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.881+0000 7fc58ed6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:26:42.892 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.881+0000 7fc58ed6b640 1 -- 172.21.15.17:0/1379868222 --> v1:172.21.15.17:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc588150c80 con 0x7fc5881422b0 2024-03-29T11:26:42.892 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.881+0000 7fc587fff640 1 -- 172.21.15.17:0/1379868222 >> v1:172.21.15.17:6789/0 conn(0x7fc5881422b0 legacy=0x7fc5880b5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-29T11:26:42.893 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.881+0000 7fc587fff640 1 -- 172.21.15.17:0/1379868222 >> v1:172.21.15.17:6789/0 conn(0x7fc5881422b0 legacy=0x7fc5880b5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:26:42.893 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.881+0000 7fc587fff640 1 --1- 172.21.15.17:0/1379868222 >> v1:172.21.15.17:6789/0 conn(0x7fc5881422b0 0x7fc5880b5780 :-1 s=OPENED pgs=743879 cs=1 l=1).handle_message read tag failed 2024-03-29T11:26:42.893 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.881+0000 7fc587fff640 1 --1- 172.21.15.17:0/1379868222 >> v1:172.21.15.17:6789/0 conn(0x7fc5881422b0 0x7fc5880b5780 :-1 s=OPENED pgs=743879 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:26:42.893 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.881+0000 7fc58ed6b640 10 client.4494.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:42.893 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.881+0000 7fc58ed6b640 10 monclient: _renew_subs 2024-03-29T11:26:42.893 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.881+0000 7fc58ed6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:26:42.893 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.881+0000 7fc58ed6b640 1 -- 172.21.15.17:0/1379868222 --> v1:172.21.15.17:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5880b12d0 con 0x7fc5881422b0 2024-03-29T11:26:42.893 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.881+0000 7fc58ed6b640 1 librados: init done 2024-03-29T11:26:42.893 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.881+0000 7fc5857fa640 10 monclient: ms_handle_reset current mon v1:172.21.15.17:6789/0 2024-03-29T11:26:42.893 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.881+0000 7fc5857fa640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:42.893 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.881+0000 7fc5857fa640 1 -- 172.21.15.17:0/1379868222 >> v1:172.21.15.17:6789/0 conn(0x7fc5881422b0 legacy=0x7fc5880b5780 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:26:42.893 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.881+0000 7fc5857fa640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:26:42.894 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.881+0000 7fc5857fa640 10 monclient(hunting): picked mon.b con 0x7fc5880b17d0 addr v1:172.21.15.94:6789/0 2024-03-29T11:26:42.894 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.882+0000 7fc5857fa640 10 monclient(hunting): picked mon.a con 0x7fc5740069c0 addr v1:172.21.15.17:6789/0 2024-03-29T11:26:42.894 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.882+0000 7fc5857fa640 10 monclient(hunting): picked mon.c con 0x7fc588142690 addr v1:172.21.15.17:6790/0 2024-03-29T11:26:42.894 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.882+0000 7fc5857fa640 1 -- 172.21.15.17:0/1379868222 --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc5880b7400 con 0x7fc5740069c0 2024-03-29T11:26:42.894 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.882+0000 7fc5857fa640 1 -- 172.21.15.17:0/1379868222 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc574004eb0 con 0x7fc5880b17d0 2024-03-29T11:26:42.894 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.882+0000 7fc5857fa640 1 -- 172.21.15.17:0/1379868222 --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc5880b7920 con 0x7fc588142690 2024-03-29T11:26:42.894 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.882+0000 7fc5857fa640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:42.894 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.882+0000 7fc5857fa640 10 client.4494.objecter ms_handle_connect 0x7fc5880b17d0 2024-03-29T11:26:42.894 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.882+0000 7fc5857fa640 10 client.4494.objecter resend_mon_ops 2024-03-29T11:26:42.894 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.882+0000 7fc5857fa640 10 client.4494.objecter ms_handle_connect 0x7fc5740069c0 2024-03-29T11:26:42.894 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.882+0000 7fc5857fa640 10 client.4494.objecter resend_mon_ops 2024-03-29T11:26:42.894 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.882+0000 7fc5857fa640 10 client.4494.objecter ms_handle_connect 0x7fc588142690 2024-03-29T11:26:42.894 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.882+0000 7fc5857fa640 10 client.4494.objecter resend_mon_ops 2024-03-29T11:26:42.894 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.882+0000 7fc5857fa640 1 -- 172.21.15.17:0/1379868222 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1569118271 0 0) 0x7fc574004eb0 con 0x7fc5880b17d0 2024-03-29T11:26:42.895 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.882+0000 7fc5857fa640 10 monclient(hunting): _negotiate 2024-03-29T11:26:42.895 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.882+0000 7fc5857fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:42.895 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.882+0000 7fc5857fa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T11:26:42.895 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.882+0000 7fc5857fa640 1 -- 172.21.15.17:0/1379868222 --> v1:172.21.15.94:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fc574012c20 con 0x7fc5880b17d0 2024-03-29T11:26:42.895 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.883+0000 7fc58ed6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:42.895 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.883+0000 7fc58ed6b640 10 monclient(hunting): _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:42.895 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.883+0000 7fc5857fa640 1 -- 172.21.15.17:0/1379868222 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1691582524 0 0) 0x7fc5880b7400 con 0x7fc5740069c0 2024-03-29T11:26:42.895 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.883+0000 7fc5857fa640 10 monclient(hunting): _negotiate 2024-03-29T11:26:42.895 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.883+0000 7fc5857fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:42.895 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.883+0000 7fc5857fa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T11:26:42.895 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.883+0000 7fc5857fa640 1 -- 172.21.15.17:0/1379868222 --> v1:172.21.15.17:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fc574004eb0 con 0x7fc5740069c0 2024-03-29T11:26:42.895 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.883+0000 7fc5857fa640 1 -- 172.21.15.17:0/1379868222 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 173399320 0 0) 0x7fc5880b7920 con 0x7fc588142690 2024-03-29T11:26:42.895 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.883+0000 7fc5857fa640 10 monclient(hunting): _negotiate 2024-03-29T11:26:42.895 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.883+0000 7fc5857fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:42.895 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.883+0000 7fc5857fa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T11:26:42.896 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.883+0000 7fc5857fa640 1 -- 172.21.15.17:0/1379868222 --> v1:172.21.15.17:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fc5880b7400 con 0x7fc588142690 2024-03-29T11:26:42.896 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.883+0000 7fc5857fa640 1 -- 172.21.15.17:0/1379868222 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 2063285762 0 0) 0x7fc574004eb0 con 0x7fc5740069c0 2024-03-29T11:26:42.896 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.883+0000 7fc5857fa640 1 -- 172.21.15.17:0/1379868222 >> v1:172.21.15.17:6790/0 conn(0x7fc588142690 legacy=0x7fc574010070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:42.896 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.883+0000 7fc5857fa640 1 -- 172.21.15.17:0/1379868222 >> v1:172.21.15.94:6789/0 conn(0x7fc5880b17d0 legacy=0x7fc574006150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:42.896 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.883+0000 7fc5857fa640 10 monclient: _finish_hunting 0 2024-03-29T11:26:42.896 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.883+0000 7fc5857fa640 1 monclient: found mon.a 2024-03-29T11:26:42.896 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.883+0000 7fc5857fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:42.896 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.883+0000 7fc5857fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:26:42.896 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.883+0000 7fc5857fa640 1 -- 172.21.15.17:0/1379868222 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7fc5880b7090 con 0x7fc5740069c0 2024-03-29T11:26:42.896 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.883+0000 7fc5857fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:26:42.896 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.883+0000 7fc5857fa640 1 -- 172.21.15.17:0/1379868222 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5880b12d0 con 0x7fc5740069c0 2024-03-29T11:26:42.897 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.883+0000 7fc5857fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:42.897 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.883+0000 7fc5857fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:26:42.897 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.883+0000 7fc5857fa640 1 -- 172.21.15.17:0/1379868222 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc574012c20 con 0x7fc5740069c0 2024-03-29T11:26:42.897 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.884+0000 7fc5857fa640 10 monclient: _finish_auth 0 2024-03-29T11:26:42.897 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.884+0000 7fc5857fa640 10 monclient: _check_auth_tickets 2024-03-29T11:26:42.897 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.884+0000 7fc5857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:42.897 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.884+0000 7fc5857fa640 1 -- 172.21.15.17:0/1379868222 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fc57c00d2d0 con 0x7fc5740069c0 2024-03-29T11:26:42.897 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.884+0000 7fc5857fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:42.897 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.884+0000 7fc5857fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:42.897 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.884+0000 7fc5857fa640 10 monclient: dump: 2024-03-29T11:26:42.897 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:26:42.897 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:26:42.897 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:42.897 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:42.897 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:42.898 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:42.898 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:26:42.898 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:26:42.898 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:26:42.898 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:42.898 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.884+0000 7fc5857fa640 1 -- 172.21.15.17:0/1379868222 <== mon.0 v1:172.21.15.17:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc57c00d4f0 con 0x7fc5740069c0 2024-03-29T11:26:42.898 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.884+0000 7fc5857fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:42.898 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.884+0000 7fc5867fc640 4 set_mon_vals no callback set 2024-03-29T11:26:42.898 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.887+0000 7fc5857fa640 1 -- 172.21.15.17:0/1379868222 <== mon.0 v1:172.21.15.17:6789/0 5 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7fc57c0234d0 con 0x7fc5740069c0 2024-03-29T11:26:42.898 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.887+0000 7fc5857fa640 1 -- 172.21.15.17:0/1379868222 <== mon.0 v1:172.21.15.17:6789/0 6 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7fc57c08d2a0 con 0x7fc5740069c0 2024-03-29T11:26:42.898 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.887+0000 7fc5857fa640 10 client.4494.objecter ms_dispatch 0x7fc588000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:42.898 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.888+0000 7fc5857fa640 3 client.4494.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:42.898 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.888+0000 7fc5857fa640 3 client.4494.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:42.898 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.888+0000 7fc58cb07640 10 monclient: get_auth_request con 0x7fc5881422b0 auth_method 0 2024-03-29T11:26:42.899 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.888+0000 7fc5857fa640 20 client.4494.objecter dump_active .. 0 homeless 2024-03-29T11:26:42.899 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.888+0000 7fc5857fa640 10 client.4494.objecter ms_handle_connect 0x7fc5881422b0 2024-03-29T11:26:42.899 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.893+0000 7fc5857fa640 1 -- 172.21.15.17:0/1379868222 <== mon.0 v1:172.21.15.17:6789/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc57c058b80 con 0x7fc5740069c0 2024-03-29T11:26:42.899 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.893+0000 7fc5857fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:42.899 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.893+0000 7fc5857fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:42.900 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.901+0000 7fc5857fa640 1 -- 172.21.15.17:0/1379868222 <== mon.0 v1:172.21.15.17:6789/0 8 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc57c05aab0 con 0x7fc5740069c0 2024-03-29T11:26:42.901 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:42.901+0000 7fc5857fa640 10 monclient: handle_mon_command_ack 1 not found 2024-03-29T11:26:43.235 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:43.248+0000 7fc58ed6b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:43.235 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:43.248+0000 7fc58ed6b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:43.235 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:43.248+0000 7fc58ed6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:26:43.236 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:43.248+0000 7fc58ed6b640 1 -- 172.21.15.17:0/1379868222 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc588075f10 con 0x7fc5740069c0 2024-03-29T11:26:43.236 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:43.250+0000 7fc5857fa640 1 -- 172.21.15.17:0/1379868222 <== mon.0 v1:172.21.15.17:6789/0 9 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7fc57c05c2d0 con 0x7fc5740069c0 2024-03-29T11:26:43.237 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:43.250+0000 7fc5857fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:43.237 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:43.250+0000 7fc5857fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:43.237 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:26:43.237 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:43.240 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:43.254+0000 7fc58ed6b640 10 librados: watch_flush enter 2024-03-29T11:26:43.240 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:43.254+0000 7fc58ed6b640 10 librados: watch_flush exit 2024-03-29T11:26:43.240 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:43.254+0000 7fc58ed6b640 20 client.4494.objecter shutdown clearing up homeless session... 2024-03-29T11:26:43.240 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:43.254+0000 7fc58ed6b640 10 client.4494.objecter successfully canceled tick 2024-03-29T11:26:43.240 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:43.254+0000 7fc58ed6b640 1 -- 172.21.15.17:0/1379868222 >> v1:172.21.15.17:6816/3236185052 conn(0x7fc5881422b0 legacy=0x7fc57407b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:43.240 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:43.254+0000 7fc58ed6b640 10 monclient: shutdown 2024-03-29T11:26:43.241 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:43.254+0000 7fc58ed6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:43.241 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:43.254+0000 7fc58ed6b640 1 -- 172.21.15.17:0/1379868222 >> v1:172.21.15.17:6789/0 conn(0x7fc5740069c0 legacy=0x7fc57400c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:43.241 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:43.254+0000 7fc58ed6b640 1 -- 172.21.15.17:0/1379868222 shutdown_connections 2024-03-29T11:26:43.241 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:43.254+0000 7fc58ed6b640 1 -- 172.21.15.17:0/1379868222 >> 172.21.15.17:0/1379868222 conn(0x7fc588066a00 msgr2=0x7fc5880b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:43.241 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:43.254+0000 7fc58ed6b640 1 -- 172.21.15.17:0/1379868222 shutdown_connections 2024-03-29T11:26:43.241 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:43.254+0000 7fc58ed6b640 1 -- 172.21.15.17:0/1379868222 wait complete. 2024-03-29T11:26:43.242 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:43.254+0000 7fc58ed6b640 1 librados: shutdown 2024-03-29T11:26:43.255 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:50.061 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:50.208 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.221+0000 7f49a215a640 10 monclient: get_monmap_and_config 2024-03-29T11:26:50.208 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.221+0000 7f49a215a640 10 monclient: build_initial_monmap 2024-03-29T11:26:50.208 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.221+0000 7f49a215a640 10 monclient: monmap: 2024-03-29T11:26:50.208 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:26:50.209 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:26:50.209 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:26:50.222304+0000 2024-03-29T11:26:50.209 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:26:50.222304+0000 2024-03-29T11:26:50.209 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:50.209 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:50.209 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:26:50.209 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:26:50.209 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:26:50.209 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:50.210 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.224+0000 7f49a215a640 1 Processor -- start 2024-03-29T11:26:50.211 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.224+0000 7f49a215a640 1 -- start start 2024-03-29T11:26:50.211 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.224+0000 7f49a215a640 10 monclient: init 2024-03-29T11:26:50.211 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.225+0000 7f49a215a640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:50.212 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.225+0000 7f49a215a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:26:50.212 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.225+0000 7f49a215a640 10 monclient(hunting): picked mon.noname-b con 0x7f499c0ae6d0 addr v1:172.21.15.17:6790/0 2024-03-29T11:26:50.212 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.225+0000 7f49a215a640 10 monclient(hunting): picked mon.noname-c con 0x7f499c0ab710 addr v1:172.21.15.94:6789/0 2024-03-29T11:26:50.212 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.225+0000 7f49a215a640 10 monclient(hunting): picked mon.noname-a con 0x7f499c13c420 addr v1:172.21.15.17:6789/0 2024-03-29T11:26:50.212 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.225+0000 7f49a215a640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f499c0add00 con 0x7f499c13c420 2024-03-29T11:26:50.212 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.225+0000 7f49a215a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f499c0ad460 con 0x7f499c0ab710 2024-03-29T11:26:50.212 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.225+0000 7f49a215a640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f499c0ac370 con 0x7f499c0ae6d0 2024-03-29T11:26:50.212 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.225+0000 7f49a215a640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:50.212 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.225+0000 7f49a215a640 10 monclient(hunting): authenticate will time out at 1316.430414s 2024-03-29T11:26:50.215 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.228+0000 7f49a1158640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f499c0ae6d0 0x7f499c13bd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:58110/0 (socket says 172.21.15.17:58110) 2024-03-29T11:26:50.215 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.228+0000 7f49a1158640 1 -- 172.21.15.17:0/4211427515 learned_addr learned my addr 172.21.15.17:0/4211427515 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:26:50.217 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.229+0000 7f4993fff640 1 -- 172.21.15.17:0/4211427515 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3359422685 0 0) 0x7f499c0ac370 con 0x7f499c0ae6d0 2024-03-29T11:26:50.217 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.229+0000 7f4993fff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:50.217 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.229+0000 7f4993fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:50.217 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.229+0000 7f4993fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:50.217 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.229+0000 7f4993fff640 10 monclient(hunting): my global_id is 4547 2024-03-29T11:26:50.217 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.229+0000 7f4993fff640 1 -- 172.21.15.17:0/4211427515 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4988003520 con 0x7f499c0ae6d0 2024-03-29T11:26:50.217 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.229+0000 7f4993fff640 1 -- 172.21.15.17:0/4211427515 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3746386667 0 0) 0x7f4988003520 con 0x7f499c0ae6d0 2024-03-29T11:26:50.217 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.229+0000 7f4993fff640 1 -- 172.21.15.17:0/4211427515 >> v1:172.21.15.94:6789/0 conn(0x7f499c0ab710 legacy=0x7f499c142170 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:26:50.217 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.229+0000 7f4993fff640 1 -- 172.21.15.17:0/4211427515 >> v1:172.21.15.17:6789/0 conn(0x7f499c13c420 legacy=0x7f499c142880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:50.221 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.231+0000 7f4993fff640 10 monclient: _finish_hunting 0 2024-03-29T11:26:50.221 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.232+0000 7f4993fff640 1 monclient: found mon.noname-b 2024-03-29T11:26:50.222 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.232+0000 7f4993fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:50.222 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.232+0000 7f4993fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.17:6790/0 2024-03-29T11:26:50.222 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.232+0000 7f4993fff640 1 -- 172.21.15.17:0/4211427515 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f499c0aa580 con 0x7f499c0ae6d0 2024-03-29T11:26:50.222 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.232+0000 7f4993fff640 10 monclient: _finish_auth 0 2024-03-29T11:26:50.222 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.232+0000 7f4993fff640 10 monclient: _check_auth_tickets 2024-03-29T11:26:50.222 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.232+0000 7f4993fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:50.222 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.232+0000 7f49a215a640 5 monclient: authenticate success, global_id 4547 2024-03-29T11:26:50.222 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.232+0000 7f49a215a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:50.222 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.232+0000 7f4993fff640 1 -- 172.21.15.17:0/4211427515 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f4994001a10 con 0x7f499c0ae6d0 2024-03-29T11:26:50.223 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.232+0000 7f4993fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:50.223 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.232+0000 7f4993fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:50.223 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.232+0000 7f4993fff640 10 monclient: dump: 2024-03-29T11:26:50.223 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:26:50.223 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:26:50.223 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:50.223 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:50.223 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:50.223 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:50.223 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:26:50.223 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:26:50.223 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:26:50.223 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:50.223 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.232+0000 7f49a215a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:50.223 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.233+0000 7f4993fff640 1 -- 172.21.15.17:0/4211427515 <== mon.2 v1:172.21.15.17:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4994001ca0 con 0x7f499c0ae6d0 2024-03-29T11:26:50.224 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.233+0000 7f4993fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:50.224 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.233+0000 7f49a215a640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:50.224 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.233+0000 7f49a215a640 4 set_mon_vals no callback set 2024-03-29T11:26:50.224 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.233+0000 7f4993fff640 1 -- 172.21.15.17:0/4211427515 <== mon.2 v1:172.21.15.17:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f4994002070 con 0x7f499c0ae6d0 2024-03-29T11:26:50.224 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.233+0000 7f4993fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:50.224 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.233+0000 7f4993fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:50.224 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.233+0000 7f4993fff640 10 monclient: dump: 2024-03-29T11:26:50.224 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:26:50.224 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:26:50.224 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:50.224 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:50.224 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:50.224 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:50.224 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:26:50.224 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:26:50.225 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:26:50.225 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:50.225 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.234+0000 7f49a215a640 10 monclient: shutdown 2024-03-29T11:26:50.225 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.234+0000 7f49a215a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:50.225 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.234+0000 7f49a215a640 1 -- 172.21.15.17:0/4211427515 >> v1:172.21.15.17:6790/0 conn(0x7f499c0ae6d0 legacy=0x7f499c13bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:50.225 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.235+0000 7f49a215a640 1 -- 172.21.15.17:0/4211427515 shutdown_connections 2024-03-29T11:26:50.226 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.235+0000 7f49a215a640 1 -- 172.21.15.17:0/4211427515 >> 172.21.15.17:0/4211427515 conn(0x7f499c0b3a30 msgr2=0x7f499c0b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:50.226 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.236+0000 7f49a215a640 1 -- 172.21.15.17:0/4211427515 shutdown_connections 2024-03-29T11:26:50.226 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.236+0000 7f49a215a640 1 -- 172.21.15.17:0/4211427515 wait complete. 2024-03-29T11:26:50.226 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.237+0000 7f49a215a640 10 monclient: build_initial_monmap 2024-03-29T11:26:50.226 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.237+0000 7f49a215a640 10 monclient: monmap: 2024-03-29T11:26:50.226 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:26:50.226 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:50.226 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:26:50.226 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:26:50.226 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:50.226 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:50.226 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:26:50.226 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:26:50.226 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:26:50.227 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:50.227 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.237+0000 7f49a215a640 1 librados: starting msgr at 2024-03-29T11:26:50.227 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.237+0000 7f49a215a640 1 librados: starting objecter 2024-03-29T11:26:50.227 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.237+0000 7f49a215a640 1 Processor -- start 2024-03-29T11:26:50.234 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.238+0000 7f49a215a640 1 -- start start 2024-03-29T11:26:50.234 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.238+0000 7f49a215a640 1 librados: setting wanted keys 2024-03-29T11:26:50.234 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.238+0000 7f49a215a640 1 librados: calling monclient init 2024-03-29T11:26:50.234 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.238+0000 7f49a215a640 10 monclient: init 2024-03-29T11:26:50.234 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.238+0000 7f49a215a640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:50.234 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.238+0000 7f49a215a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:26:50.234 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.238+0000 7f49a215a640 10 monclient(hunting): picked mon.noname-a con 0x7f499c0ab710 addr v1:172.21.15.17:6789/0 2024-03-29T11:26:50.235 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.238+0000 7f49a215a640 10 monclient(hunting): picked mon.noname-c con 0x7f499c0ae6d0 addr v1:172.21.15.17:6790/0 2024-03-29T11:26:50.235 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.238+0000 7f49a215a640 10 monclient(hunting): picked mon.noname-b con 0x7f499c13c420 addr v1:172.21.15.94:6789/0 2024-03-29T11:26:50.235 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.238+0000 7f49a215a640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f499c13f670 con 0x7f499c0ab710 2024-03-29T11:26:50.235 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.238+0000 7f49a215a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f499c06e410 con 0x7f499c13c420 2024-03-29T11:26:50.235 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.238+0000 7f49a215a640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f499c13eaf0 con 0x7f499c0ae6d0 2024-03-29T11:26:50.235 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.238+0000 7f49a215a640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:50.235 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.238+0000 7f49a215a640 10 monclient(hunting): authenticate will time out at 1316.443760s 2024-03-29T11:26:50.235 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.242+0000 7f49a1158640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f499c0ab710 0x7f499c13edf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:44243/0 (socket says 172.21.15.17:44243) 2024-03-29T11:26:50.235 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.242+0000 7f49a1158640 1 -- 172.21.15.17:0/82728158 learned_addr learned my addr 172.21.15.17:0/82728158 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:26:50.235 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.242+0000 7f4991ffb640 10 client.?.objecter ms_handle_connect 0x7f499c0ab710 2024-03-29T11:26:50.235 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.242+0000 7f4991ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:50.235 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.242+0000 7f4991ffb640 1 -- 172.21.15.17:0/82728158 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 716875603 0 0) 0x7f499c13f670 con 0x7f499c0ab710 2024-03-29T11:26:50.235 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.242+0000 7f4991ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:50.236 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.242+0000 7f4991ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:50.236 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.242+0000 7f4991ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:50.236 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.242+0000 7f4991ffb640 10 monclient(hunting): my global_id is 4497 2024-03-29T11:26:50.236 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.242+0000 7f4991ffb640 1 -- 172.21.15.17:0/82728158 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4980004eb0 con 0x7f499c0ab710 2024-03-29T11:26:50.236 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.243+0000 7f4991ffb640 10 client.?.objecter ms_handle_connect 0x7f499c0ae6d0 2024-03-29T11:26:50.236 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.243+0000 7f4991ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:50.236 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.243+0000 7f4991ffb640 1 -- 172.21.15.17:0/82728158 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 330919129 0 0) 0x7f4980004eb0 con 0x7f499c0ab710 2024-03-29T11:26:50.236 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.243+0000 7f4991ffb640 1 -- 172.21.15.17:0/82728158 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f499c13f670 con 0x7f499c0ab710 2024-03-29T11:26:50.236 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.243+0000 7f4991ffb640 1 -- 172.21.15.17:0/82728158 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3712241716 0 0) 0x7f499c13eaf0 con 0x7f499c0ae6d0 2024-03-29T11:26:50.236 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.243+0000 7f4991ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:50.236 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.243+0000 7f4991ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:50.236 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.243+0000 7f4991ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:50.236 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.243+0000 7f4991ffb640 10 monclient(hunting): my global_id is 4550 2024-03-29T11:26:50.236 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.243+0000 7f4991ffb640 1 -- 172.21.15.17:0/82728158 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4980004eb0 con 0x7f499c0ae6d0 2024-03-29T11:26:50.237 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.243+0000 7f4991ffb640 1 -- 172.21.15.17:0/82728158 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f4994001990 con 0x7f499c0ab710 2024-03-29T11:26:50.237 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.243+0000 7f4991ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:50.237 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.244+0000 7f4991ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:50.237 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.244+0000 7f4991ffb640 10 monclient(hunting): dump: 2024-03-29T11:26:50.237 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:26:50.237 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:26:50.237 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:50.237 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:50.237 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:50.237 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:50.237 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:26:50.237 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:26:50.237 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:26:50.238 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:50.238 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.244+0000 7f4991ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:50.238 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.244+0000 7f4991ffb640 1 -- 172.21.15.17:0/82728158 <== mon.0 v1:172.21.15.17:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3022833713 0 0) 0x7f499c13f670 con 0x7f499c0ab710 2024-03-29T11:26:50.238 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.244+0000 7f4991ffb640 1 -- 172.21.15.17:0/82728158 >> v1:172.21.15.17:6790/0 conn(0x7f499c0ae6d0 legacy=0x7f499c071f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:50.238 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.244+0000 7f4991ffb640 1 -- 172.21.15.17:0/82728158 >> v1:172.21.15.94:6789/0 conn(0x7f499c13c420 legacy=0x7f499c1d2be0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:26:50.238 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.245+0000 7f4991ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:26:50.238 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.245+0000 7f4991ffb640 1 monclient: found mon.a 2024-03-29T11:26:50.238 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.245+0000 7f4991ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:50.238 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.245+0000 7f4991ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:26:50.238 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.245+0000 7f4991ffb640 1 -- 172.21.15.17:0/82728158 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f499c072680 con 0x7f499c0ab710 2024-03-29T11:26:50.238 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.245+0000 7f4991ffb640 10 monclient: _finish_auth 0 2024-03-29T11:26:50.239 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.245+0000 7f4991ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:26:50.239 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.245+0000 7f4991ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:50.239 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.247+0000 7f49a215a640 5 monclient: authenticate success, global_id 4497 2024-03-29T11:26:50.239 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.247+0000 7f49a215a640 10 monclient: _renew_subs 2024-03-29T11:26:50.239 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.247+0000 7f49a215a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:26:50.239 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.247+0000 7f49a215a640 1 -- 172.21.15.17:0/82728158 --> v1:172.21.15.17:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f499c072930 con 0x7f499c0ab710 2024-03-29T11:26:50.239 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.247+0000 7f49a215a640 10 client.4497.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:50.239 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.247+0000 7f49a215a640 10 monclient: _renew_subs 2024-03-29T11:26:50.239 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.247+0000 7f49a215a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:26:50.240 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.247+0000 7f49a215a640 1 -- 172.21.15.17:0/82728158 --> v1:172.21.15.17:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f499c0b2d30 con 0x7f499c0ab710 2024-03-29T11:26:50.240 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.247+0000 7f49a215a640 1 librados: init done 2024-03-29T11:26:50.242 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.249+0000 7f4991ffb640 1 -- 172.21.15.17:0/82728158 <== mon.0 v1:172.21.15.17:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4994005020 con 0x7f499c0ab710 2024-03-29T11:26:50.242 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.249+0000 7f4991ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:50.242 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.249+0000 7f4991ffb640 1 -- 172.21.15.17:0/82728158 <== mon.0 v1:172.21.15.17:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f4994005430 con 0x7f499c0ab710 2024-03-29T11:26:50.242 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.249+0000 7f4991ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:50.243 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.249+0000 7f4991ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:50.243 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.249+0000 7f4991ffb640 10 monclient: dump: 2024-03-29T11:26:50.243 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:26:50.243 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:26:50.243 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:50.243 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:50.243 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:50.243 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:50.243 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:26:50.243 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:26:50.243 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:26:50.243 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:50.243 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.249+0000 7f49937fe640 4 set_mon_vals no callback set 2024-03-29T11:26:50.243 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.250+0000 7f49a215a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:50.243 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.250+0000 7f49a215a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:50.244 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.251+0000 7f49a215a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:26:50.244 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.251+0000 7f4991ffb640 1 -- 172.21.15.17:0/82728158 <== mon.0 v1:172.21.15.17:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f4994005590 con 0x7f499c0ab710 2024-03-29T11:26:50.244 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.251+0000 7f49a215a640 1 -- 172.21.15.17:0/82728158 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f497c005080 con 0x7f499c0ab710 2024-03-29T11:26:50.244 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.251+0000 7f4991ffb640 1 -- 172.21.15.17:0/82728158 <== mon.0 v1:172.21.15.17:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f4994085020 con 0x7f499c0ab710 2024-03-29T11:26:50.244 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.251+0000 7f4991ffb640 10 client.4497.objecter ms_dispatch 0x7f499c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:50.244 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.251+0000 7f4991ffb640 3 client.4497.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:50.244 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.251+0000 7f4991ffb640 3 client.4497.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:50.244 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.251+0000 7f4991ffb640 20 client.4497.objecter dump_active .. 0 homeless 2024-03-29T11:26:50.244 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.252+0000 7f49a0957640 10 monclient: get_auth_request con 0x7f498006deb0 auth_method 0 2024-03-29T11:26:50.244 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.252+0000 7f4991ffb640 10 client.4497.objecter ms_handle_connect 0x7f498006deb0 2024-03-29T11:26:50.247 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.260+0000 7f4991ffb640 1 -- 172.21.15.17:0/82728158 <== mon.0 v1:172.21.15.17:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4994050900 con 0x7f499c0ab710 2024-03-29T11:26:50.247 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.260+0000 7f4991ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:50.247 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.260+0000 7f4991ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:50.579 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.592+0000 7f49a215a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:50.579 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.592+0000 7f49a215a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:50.579 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.592+0000 7f49a215a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:26:50.579 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.592+0000 7f49a215a640 1 -- 172.21.15.17:0/82728158 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f497c005330 con 0x7f499c0ab710 2024-03-29T11:26:50.580 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.593+0000 7f4991ffb640 1 -- 172.21.15.17:0/82728158 <== mon.0 v1:172.21.15.17:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f4994052830 con 0x7f499c0ab710 2024-03-29T11:26:50.580 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.593+0000 7f4991ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:50.580 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.593+0000 7f4991ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:50.580 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:26:50.581 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:50.583 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.597+0000 7f49a215a640 10 librados: watch_flush enter 2024-03-29T11:26:50.584 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.597+0000 7f49a215a640 10 librados: watch_flush exit 2024-03-29T11:26:50.584 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.597+0000 7f49a215a640 20 client.4497.objecter shutdown clearing up homeless session... 2024-03-29T11:26:50.584 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.597+0000 7f49a215a640 10 client.4497.objecter successfully canceled tick 2024-03-29T11:26:50.584 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.597+0000 7f49a215a640 1 -- 172.21.15.17:0/82728158 >> v1:172.21.15.17:6816/3236185052 conn(0x7f498006deb0 legacy=0x7f4980070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:50.584 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.597+0000 7f49a215a640 10 monclient: shutdown 2024-03-29T11:26:50.584 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.597+0000 7f49a215a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:50.584 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.597+0000 7f49a215a640 1 -- 172.21.15.17:0/82728158 >> v1:172.21.15.17:6789/0 conn(0x7f499c0ab710 legacy=0x7f499c13edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:50.585 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.598+0000 7f49a215a640 1 -- 172.21.15.17:0/82728158 shutdown_connections 2024-03-29T11:26:50.585 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.598+0000 7f49a215a640 1 -- 172.21.15.17:0/82728158 >> 172.21.15.17:0/82728158 conn(0x7f499c0b3a30 msgr2=0x7f499c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:50.585 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.598+0000 7f49a215a640 1 -- 172.21.15.17:0/82728158 shutdown_connections 2024-03-29T11:26:50.585 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.598+0000 7f49a215a640 1 -- 172.21.15.17:0/82728158 wait complete. 2024-03-29T11:26:50.585 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:50.598+0000 7f49a215a640 1 librados: shutdown 2024-03-29T11:26:50.605 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:57.412 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:57.560 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.573+0000 7f63ccd64640 10 monclient: get_monmap_and_config 2024-03-29T11:26:57.560 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.573+0000 7f63ccd64640 10 monclient: build_initial_monmap 2024-03-29T11:26:57.561 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.573+0000 7f63ccd64640 10 monclient: monmap: 2024-03-29T11:26:57.561 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:26:57.561 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:26:57.561 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:26:57.574152+0000 2024-03-29T11:26:57.561 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:26:57.574152+0000 2024-03-29T11:26:57.561 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:57.561 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:57.561 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:26:57.561 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:26:57.561 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:26:57.561 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:57.564 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.577+0000 7f63ccd64640 1 Processor -- start 2024-03-29T11:26:57.564 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.577+0000 7f63ccd64640 1 -- start start 2024-03-29T11:26:57.564 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.577+0000 7f63ccd64640 10 monclient: init 2024-03-29T11:26:57.566 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.578+0000 7f63ccd64640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:57.566 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.578+0000 7f63ccd64640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:26:57.566 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.578+0000 7f63ccd64640 10 monclient(hunting): picked mon.noname-c con 0x7f63c0006b60 addr v1:172.21.15.94:6789/0 2024-03-29T11:26:57.566 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.578+0000 7f63ccd64640 10 monclient(hunting): picked mon.noname-a con 0x7f63c002c5c0 addr v1:172.21.15.17:6789/0 2024-03-29T11:26:57.566 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.578+0000 7f63ccd64640 10 monclient(hunting): picked mon.noname-b con 0x7f63c00c2c20 addr v1:172.21.15.17:6790/0 2024-03-29T11:26:57.566 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.578+0000 7f63ccd64640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c002be50 con 0x7f63c002c5c0 2024-03-29T11:26:57.566 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.578+0000 7f63ccd64640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c002c130 con 0x7f63c0006b60 2024-03-29T11:26:57.566 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.578+0000 7f63ccd64640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c002e310 con 0x7f63c00c2c20 2024-03-29T11:26:57.566 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.578+0000 7f63ccd64640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:57.566 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.578+0000 7f63ccd64640 10 monclient(hunting): authenticate will time out at 1323.783330s 2024-03-29T11:26:57.569 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.582+0000 7f63c6ffd640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f63c002c5c0 0x7f63c00c2510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:58607/0 (socket says 172.21.15.17:58607) 2024-03-29T11:26:57.569 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.582+0000 7f63c6ffd640 1 -- 172.21.15.17:0/2123194049 learned_addr learned my addr 172.21.15.17:0/2123194049 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:26:57.569 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.582+0000 7f63c67fc640 1 -- 172.21.15.17:0/2123194049 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3909694520 0 0) 0x7f63c002be50 con 0x7f63c002c5c0 2024-03-29T11:26:57.569 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.582+0000 7f63c67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:57.571 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.582+0000 7f63c67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:57.571 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.582+0000 7f63c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:57.571 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.582+0000 7f63c67fc640 10 monclient(hunting): my global_id is 4500 2024-03-29T11:26:57.571 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.582+0000 7f63c67fc640 1 -- 172.21.15.17:0/2123194049 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63b8003520 con 0x7f63c002c5c0 2024-03-29T11:26:57.571 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.582+0000 7f63c67fc640 1 -- 172.21.15.17:0/2123194049 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2191767440 0 0) 0x7f63c002e310 con 0x7f63c00c2c20 2024-03-29T11:26:57.571 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.582+0000 7f63c67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:57.571 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.582+0000 7f63c67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:57.572 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.582+0000 7f63c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:57.572 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.582+0000 7f63c67fc640 10 monclient(hunting): my global_id is 4553 2024-03-29T11:26:57.572 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.582+0000 7f63c67fc640 1 -- 172.21.15.17:0/2123194049 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63c002be50 con 0x7f63c00c2c20 2024-03-29T11:26:57.572 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.583+0000 7f63c67fc640 1 -- 172.21.15.17:0/2123194049 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1699924701 0 0) 0x7f63c002be50 con 0x7f63c00c2c20 2024-03-29T11:26:57.572 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.583+0000 7f63c67fc640 1 -- 172.21.15.17:0/2123194049 >> v1:172.21.15.94:6789/0 conn(0x7f63c0006b60 legacy=0x7f63c00c8af0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:26:57.572 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.583+0000 7f63c67fc640 1 -- 172.21.15.17:0/2123194049 >> v1:172.21.15.17:6789/0 conn(0x7f63c002c5c0 legacy=0x7f63c00c2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:57.572 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.583+0000 7f63c67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:26:57.572 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.583+0000 7f63c67fc640 1 monclient: found mon.noname-b 2024-03-29T11:26:57.572 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.583+0000 7f63c67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:57.572 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.583+0000 7f63c67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.17:6790/0 2024-03-29T11:26:57.572 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.583+0000 7f63c67fc640 1 -- 172.21.15.17:0/2123194049 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c002e5f0 con 0x7f63c00c2c20 2024-03-29T11:26:57.572 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.583+0000 7f63c67fc640 10 monclient: _finish_auth 0 2024-03-29T11:26:57.572 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.583+0000 7f63c67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:26:57.572 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.583+0000 7f63c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:57.572 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.583+0000 7f63c67fc640 1 -- 172.21.15.17:0/2123194049 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f63c804f730 con 0x7f63c00c2c20 2024-03-29T11:26:57.573 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.583+0000 7f63c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:57.573 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.583+0000 7f63c67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:57.573 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.583+0000 7f63c67fc640 10 monclient: dump: 2024-03-29T11:26:57.573 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:26:57.573 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:26:57.573 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:57.573 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:57.573 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:57.573 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:57.573 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:26:57.574 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:26:57.574 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:26:57.574 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:57.575 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.586+0000 7f63ccd64640 5 monclient: authenticate success, global_id 4553 2024-03-29T11:26:57.575 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.586+0000 7f63ccd64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:57.575 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.586+0000 7f63c67fc640 1 -- 172.21.15.17:0/2123194049 <== mon.2 v1:172.21.15.17:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63c8075f10 con 0x7f63c00c2c20 2024-03-29T11:26:57.575 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.586+0000 7f63c67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:57.575 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.586+0000 7f63ccd64640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:57.575 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.586+0000 7f63ccd64640 4 set_mon_vals no callback set 2024-03-29T11:26:57.575 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.586+0000 7f63c67fc640 1 -- 172.21.15.17:0/2123194049 <== mon.2 v1:172.21.15.17:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f63c80758b0 con 0x7f63c00c2c20 2024-03-29T11:26:57.575 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.586+0000 7f63c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:57.575 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.586+0000 7f63c67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:57.576 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.586+0000 7f63c67fc640 10 monclient: dump: 2024-03-29T11:26:57.576 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:26:57.576 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:26:57.576 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:57.576 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:57.576 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:57.576 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:57.576 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:26:57.576 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:26:57.576 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:26:57.576 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:57.576 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.587+0000 7f63ccd64640 10 monclient: shutdown 2024-03-29T11:26:57.576 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.587+0000 7f63ccd64640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:57.576 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.587+0000 7f63ccd64640 1 -- 172.21.15.17:0/2123194049 >> v1:172.21.15.17:6790/0 conn(0x7f63c00c2c20 legacy=0x7f63c00d0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:57.576 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.587+0000 7f63ccd64640 1 -- 172.21.15.17:0/2123194049 shutdown_connections 2024-03-29T11:26:57.577 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.587+0000 7f63ccd64640 1 -- 172.21.15.17:0/2123194049 >> 172.21.15.17:0/2123194049 conn(0x7f63c00058f0 msgr2=0x7f63c0005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:57.577 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.589+0000 7f63ccd64640 1 -- 172.21.15.17:0/2123194049 shutdown_connections 2024-03-29T11:26:57.577 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.589+0000 7f63ccd64640 1 -- 172.21.15.17:0/2123194049 wait complete. 2024-03-29T11:26:57.578 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.590+0000 7f63ccd64640 10 monclient: build_initial_monmap 2024-03-29T11:26:57.578 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.590+0000 7f63ccd64640 10 monclient: monmap: 2024-03-29T11:26:57.578 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:26:57.578 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:57.578 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:26:57.579 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:26:57.579 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:57.579 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:57.579 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:26:57.579 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:26:57.579 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:26:57.579 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:57.580 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.591+0000 7f63ccd64640 1 librados: starting msgr at 2024-03-29T11:26:57.580 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.591+0000 7f63ccd64640 1 librados: starting objecter 2024-03-29T11:26:57.581 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.591+0000 7f63ccd64640 1 Processor -- start 2024-03-29T11:26:57.581 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.591+0000 7f63ccd64640 1 -- start start 2024-03-29T11:26:57.581 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.591+0000 7f63ccd64640 1 librados: setting wanted keys 2024-03-29T11:26:57.581 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.591+0000 7f63ccd64640 1 librados: calling monclient init 2024-03-29T11:26:57.581 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.591+0000 7f63ccd64640 10 monclient: init 2024-03-29T11:26:57.582 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.592+0000 7f63ccd64640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:57.582 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.592+0000 7f63ccd64640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:26:57.582 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.592+0000 7f63ccd64640 10 monclient(hunting): picked mon.noname-c con 0x7f63c0006b60 addr v1:172.21.15.17:6790/0 2024-03-29T11:26:57.582 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.592+0000 7f63ccd64640 10 monclient(hunting): picked mon.noname-a con 0x7f63c002c5c0 addr v1:172.21.15.17:6789/0 2024-03-29T11:26:57.582 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.592+0000 7f63ccd64640 10 monclient(hunting): picked mon.noname-b con 0x7f63c00c2c20 addr v1:172.21.15.94:6789/0 2024-03-29T11:26:57.582 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.592+0000 7f63ccd64640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c00bda60 con 0x7f63c002c5c0 2024-03-29T11:26:57.582 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.592+0000 7f63c77fe640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f63c0006b60 0x7f63c01587b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:40964/0 (socket says 172.21.15.17:40964) 2024-03-29T11:26:57.583 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.592+0000 7f63c77fe640 1 -- 172.21.15.17:0/4125678896 learned_addr learned my addr 172.21.15.17:0/4125678896 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:26:57.583 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.593+0000 7f63b3fff640 10 client.?.objecter ms_handle_connect 0x7f63c0006b60 2024-03-29T11:26:57.583 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.593+0000 7f63b3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:57.583 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.595+0000 7f63ccd64640 1 -- 172.21.15.17:0/4125678896 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c002fc50 con 0x7f63c00c2c20 2024-03-29T11:26:57.583 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.595+0000 7f63ccd64640 1 -- 172.21.15.17:0/4125678896 --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c0158290 con 0x7f63c0006b60 2024-03-29T11:26:57.583 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.595+0000 7f63ccd64640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:57.583 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.595+0000 7f63ccd64640 10 monclient(hunting): authenticate will time out at 1323.800501s 2024-03-29T11:26:57.583 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.596+0000 7f63b3fff640 1 -- 172.21.15.17:0/4125678896 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3066680124 0 0) 0x7f63c0158290 con 0x7f63c0006b60 2024-03-29T11:26:57.583 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.596+0000 7f63b3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:57.583 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.596+0000 7f63b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:57.583 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.596+0000 7f63b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:57.584 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.596+0000 7f63b3fff640 10 monclient(hunting): my global_id is 4556 2024-03-29T11:26:57.584 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.596+0000 7f63b3fff640 1 -- 172.21.15.17:0/4125678896 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63ac004eb0 con 0x7f63c0006b60 2024-03-29T11:26:57.584 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.597+0000 7f63b3fff640 1 -- 172.21.15.17:0/4125678896 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3026444308 0 0) 0x7f63ac004eb0 con 0x7f63c0006b60 2024-03-29T11:26:57.584 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.597+0000 7f63b3fff640 1 -- 172.21.15.17:0/4125678896 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63ac005300 con 0x7f63c0006b60 2024-03-29T11:26:57.585 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.597+0000 7f63b3fff640 1 -- 172.21.15.17:0/4125678896 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f63bc001830 con 0x7f63c0006b60 2024-03-29T11:26:57.585 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.597+0000 7f63b3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:57.585 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.597+0000 7f63b3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:57.585 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.597+0000 7f63b3fff640 10 monclient(hunting): dump: 2024-03-29T11:26:57.585 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:26:57.585 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:26:57.585 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:57.585 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:57.585 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:57.585 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:57.585 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:26:57.586 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:26:57.586 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:26:57.586 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:57.586 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.597+0000 7f63b3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:57.586 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.597+0000 7f63b3fff640 1 -- 172.21.15.17:0/4125678896 <== mon.2 v1:172.21.15.17:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 614265206 0 0) 0x7f63ac005300 con 0x7f63c0006b60 2024-03-29T11:26:57.586 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.597+0000 7f63b3fff640 1 -- 172.21.15.17:0/4125678896 >> v1:172.21.15.94:6789/0 conn(0x7f63c00c2c20 legacy=0x7f63c00bd350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:26:57.586 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.598+0000 7f63b3fff640 1 -- 172.21.15.17:0/4125678896 >> v1:172.21.15.17:6789/0 conn(0x7f63c002c5c0 legacy=0x7f63c00bcc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:57.587 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.598+0000 7f63b3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:26:57.587 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.598+0000 7f63b3fff640 1 monclient: found mon.c 2024-03-29T11:26:57.587 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.598+0000 7f63b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:57.587 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.598+0000 7f63b3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:26:57.587 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.598+0000 7f63b3fff640 1 -- 172.21.15.17:0/4125678896 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c0158ec0 con 0x7f63c0006b60 2024-03-29T11:26:57.587 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.598+0000 7f63b3fff640 10 monclient: _finish_auth 0 2024-03-29T11:26:57.587 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.598+0000 7f63b3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:26:57.587 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.598+0000 7f63b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:57.587 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.598+0000 7f63b3fff640 10 client.?.objecter ms_handle_connect 0x7f63c002c5c0 2024-03-29T11:26:57.587 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.598+0000 7f63b3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:57.587 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.598+0000 7f63ccd64640 5 monclient: authenticate success, global_id 4556 2024-03-29T11:26:57.587 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.598+0000 7f63ccd64640 10 monclient: _renew_subs 2024-03-29T11:26:57.587 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.598+0000 7f63ccd64640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:26:57.587 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.598+0000 7f63ccd64640 1 -- 172.21.15.17:0/4125678896 --> v1:172.21.15.17:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63c002eaf0 con 0x7f63c0006b60 2024-03-29T11:26:57.588 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.598+0000 7f63ccd64640 10 client.4556.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:57.588 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.598+0000 7f63ccd64640 10 monclient: _renew_subs 2024-03-29T11:26:57.588 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.598+0000 7f63ccd64640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:26:57.588 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.598+0000 7f63ccd64640 1 -- 172.21.15.17:0/4125678896 --> v1:172.21.15.17:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f63c00c3550 con 0x7f63c0006b60 2024-03-29T11:26:57.588 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.598+0000 7f63ccd64640 1 librados: init done 2024-03-29T11:26:57.588 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.599+0000 7f63ccd64640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:57.588 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.599+0000 7f63ccd64640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:57.588 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.600+0000 7f63ccd64640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:26:57.588 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.600+0000 7f63b3fff640 1 -- 172.21.15.17:0/4125678896 <== mon.2 v1:172.21.15.17:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63bc001e00 con 0x7f63c0006b60 2024-03-29T11:26:57.588 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.600+0000 7f63ccd64640 1 -- 172.21.15.17:0/4125678896 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63c01590b0 con 0x7f63c0006b60 2024-03-29T11:26:57.589 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.600+0000 7f63b3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:57.589 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.600+0000 7f63c57fa640 4 set_mon_vals no callback set 2024-03-29T11:26:57.589 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.600+0000 7f63b3fff640 1 -- 172.21.15.17:0/4125678896 <== mon.2 v1:172.21.15.17:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f63bc001830 con 0x7f63c0006b60 2024-03-29T11:26:57.589 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.600+0000 7f63b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:57.589 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.600+0000 7f63b3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:57.589 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.600+0000 7f63b3fff640 10 monclient: dump: 2024-03-29T11:26:57.589 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:26:57.589 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:26:57.589 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:57.589 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:26:57.589 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:57.589 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:26:57.589 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:26:57.589 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:26:57.589 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:26:57.590 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:26:57.595 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.601+0000 7f63b3fff640 1 -- 172.21.15.17:0/4125678896 <== mon.2 v1:172.21.15.17:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f63bc01a8b0 con 0x7f63c0006b60 2024-03-29T11:26:57.595 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.602+0000 7f63b3fff640 1 -- 172.21.15.17:0/4125678896 <== mon.2 v1:172.21.15.17:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f63bc054460 con 0x7f63c0006b60 2024-03-29T11:26:57.595 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.602+0000 7f63b3fff640 10 client.4556.objecter ms_dispatch 0x7f63c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:57.595 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.602+0000 7f63b3fff640 3 client.4556.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:57.595 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.602+0000 7f63b3fff640 3 client.4556.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:57.595 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.602+0000 7f63b3fff640 20 client.4556.objecter dump_active .. 0 homeless 2024-03-29T11:26:57.595 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.603+0000 7f63c6ffd640 10 monclient: get_auth_request con 0x7f63ac06e140 auth_method 0 2024-03-29T11:26:57.595 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.603+0000 7f63b3fff640 10 client.4556.objecter ms_handle_connect 0x7f63ac06e140 2024-03-29T11:26:57.597 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.610+0000 7f63b3fff640 1 -- 172.21.15.17:0/4125678896 <== mon.2 v1:172.21.15.17:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f63bc050700 con 0x7f63c0006b60 2024-03-29T11:26:57.597 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.610+0000 7f63b3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:57.597 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.610+0000 7f63b3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:57.934 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.947+0000 7f63ccd64640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:57.934 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.947+0000 7f63ccd64640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:57.934 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.947+0000 7f63ccd64640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:26:57.935 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.947+0000 7f63ccd64640 1 -- 172.21.15.17:0/4125678896 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63c0159050 con 0x7f63c0006b60 2024-03-29T11:26:57.938 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.951+0000 7f63b3fff640 1 -- 172.21.15.17:0/4125678896 <== mon.2 v1:172.21.15.17:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f63bc0508a0 con 0x7f63c0006b60 2024-03-29T11:26:57.938 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.951+0000 7f63b3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:57.938 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.951+0000 7f63b3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:57.939 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:26:57.939 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:57.942 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.954+0000 7f63ccd64640 10 librados: watch_flush enter 2024-03-29T11:26:57.942 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.954+0000 7f63ccd64640 10 librados: watch_flush exit 2024-03-29T11:26:57.942 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.954+0000 7f63ccd64640 20 client.4556.objecter shutdown clearing up homeless session... 2024-03-29T11:26:57.942 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.954+0000 7f63ccd64640 10 client.4556.objecter successfully canceled tick 2024-03-29T11:26:57.942 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.954+0000 7f63ccd64640 1 -- 172.21.15.17:0/4125678896 >> v1:172.21.15.17:6816/3236185052 conn(0x7f63ac06e140 legacy=0x7f63ac070600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:57.942 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.954+0000 7f63ccd64640 10 monclient: shutdown 2024-03-29T11:26:57.942 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.954+0000 7f63ccd64640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:57.942 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.954+0000 7f63ccd64640 1 -- 172.21.15.17:0/4125678896 >> v1:172.21.15.17:6790/0 conn(0x7f63c0006b60 legacy=0x7f63c01587b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:57.942 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.955+0000 7f63ccd64640 1 -- 172.21.15.17:0/4125678896 shutdown_connections 2024-03-29T11:26:57.942 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.955+0000 7f63ccd64640 1 -- 172.21.15.17:0/4125678896 >> 172.21.15.17:0/4125678896 conn(0x7f63c00058f0 msgr2=0x7f63c00c22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:57.944 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.957+0000 7f63ccd64640 1 -- 172.21.15.17:0/4125678896 shutdown_connections 2024-03-29T11:26:57.944 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.957+0000 7f63ccd64640 1 -- 172.21.15.17:0/4125678896 wait complete. 2024-03-29T11:26:57.944 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:26:57.957+0000 7f63ccd64640 1 librados: shutdown 2024-03-29T11:26:57.965 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:04.773 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:04.926 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.939+0000 7f1b15c95640 10 monclient: get_monmap_and_config 2024-03-29T11:27:04.927 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.939+0000 7f1b15c95640 10 monclient: build_initial_monmap 2024-03-29T11:27:04.927 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.939+0000 7f1b15c95640 10 monclient: monmap: 2024-03-29T11:27:04.927 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:27:04.927 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:27:04.927 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:27:04.940356+0000 2024-03-29T11:27:04.927 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:27:04.940356+0000 2024-03-29T11:27:04.927 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:04.927 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:04.927 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:27:04.927 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:27:04.927 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:27:04.927 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:04.929 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.941+0000 7f1b15c95640 1 Processor -- start 2024-03-29T11:27:04.929 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.942+0000 7f1b15c95640 1 -- start start 2024-03-29T11:27:04.929 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.942+0000 7f1b15c95640 10 monclient: init 2024-03-29T11:27:04.930 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.943+0000 7f1b15c95640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:04.930 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.943+0000 7f1b15c95640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:27:04.930 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.943+0000 7f1b15c95640 10 monclient(hunting): picked mon.noname-b con 0x7f1b100b0770 addr v1:172.21.15.17:6790/0 2024-03-29T11:27:04.930 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.943+0000 7f1b15c95640 10 monclient(hunting): picked mon.noname-c con 0x7f1b1014a700 addr v1:172.21.15.94:6789/0 2024-03-29T11:27:04.930 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.943+0000 7f1b15c95640 10 monclient(hunting): picked mon.noname-a con 0x7f1b100ac5a0 addr v1:172.21.15.17:6789/0 2024-03-29T11:27:04.930 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.943+0000 7f1b15c95640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b100aba70 con 0x7f1b100ac5a0 2024-03-29T11:27:04.930 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.943+0000 7f1b15c95640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b100aecb0 con 0x7f1b1014a700 2024-03-29T11:27:04.930 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.943+0000 7f1b15c95640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b100af150 con 0x7f1b100b0770 2024-03-29T11:27:04.930 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.943+0000 7f1b15c95640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:04.931 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.943+0000 7f1b15c95640 10 monclient(hunting): authenticate will time out at 1331.148094s 2024-03-29T11:27:04.931 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.944+0000 7f1b0f7fe640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f1b100b0770 0x7f1b10149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:59988/0 (socket says 172.21.15.17:59988) 2024-03-29T11:27:04.931 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.944+0000 7f1b0f7fe640 1 -- 172.21.15.17:0/2710441017 learned_addr learned my addr 172.21.15.17:0/2710441017 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:27:04.934 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.946+0000 7f1b0e7fc640 1 -- 172.21.15.17:0/2710441017 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1386753784 0 0) 0x7f1b100af150 con 0x7f1b100b0770 2024-03-29T11:27:04.934 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.946+0000 7f1b0e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:27:04.934 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.946+0000 7f1b0e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:04.934 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.946+0000 7f1b0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:04.934 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.946+0000 7f1b0e7fc640 10 monclient(hunting): my global_id is 4559 2024-03-29T11:27:04.934 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.946+0000 7f1b0e7fc640 1 -- 172.21.15.17:0/2710441017 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1af8003520 con 0x7f1b100b0770 2024-03-29T11:27:04.934 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.947+0000 7f1b0e7fc640 1 -- 172.21.15.17:0/2710441017 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1957520596 0 0) 0x7f1af8003520 con 0x7f1b100b0770 2024-03-29T11:27:04.935 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.947+0000 7f1b0e7fc640 1 -- 172.21.15.17:0/2710441017 >> v1:172.21.15.94:6789/0 conn(0x7f1b1014a700 legacy=0x7f1b100abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:04.935 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.947+0000 7f1b0e7fc640 1 -- 172.21.15.17:0/2710441017 >> v1:172.21.15.17:6789/0 conn(0x7f1b100ac5a0 legacy=0x7f1b10151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:27:04.935 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.947+0000 7f1b0e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:27:04.935 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.947+0000 7f1b0e7fc640 1 monclient: found mon.noname-b 2024-03-29T11:27:04.935 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.947+0000 7f1b0e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:04.935 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.947+0000 7f1b0e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.17:6790/0 2024-03-29T11:27:04.935 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.947+0000 7f1b0e7fc640 1 -- 172.21.15.17:0/2710441017 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b100ac9d0 con 0x7f1b100b0770 2024-03-29T11:27:04.935 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.947+0000 7f1b0e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:27:04.935 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.947+0000 7f1b0e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:27:04.935 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.947+0000 7f1b0e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:04.935 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.947+0000 7f1b0e7fc640 1 -- 172.21.15.17:0/2710441017 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f1b04001690 con 0x7f1b100b0770 2024-03-29T11:27:04.935 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.947+0000 7f1b0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:04.935 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.947+0000 7f1b0e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:04.935 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.947+0000 7f1b0e7fc640 10 monclient: dump: 2024-03-29T11:27:04.935 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:27:04.935 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:27:04.936 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:04.936 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:04.936 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:04.936 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:04.936 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:27:04.936 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:27:04.936 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:27:04.936 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:04.937 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.947+0000 7f1b15c95640 5 monclient: authenticate success, global_id 4559 2024-03-29T11:27:04.937 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.947+0000 7f1b15c95640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:04.937 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.949+0000 7f1b0e7fc640 1 -- 172.21.15.17:0/2710441017 <== mon.2 v1:172.21.15.17:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b04001860 con 0x7f1b100b0770 2024-03-29T11:27:04.937 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.949+0000 7f1b0e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:04.939 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.949+0000 7f1b15c95640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:04.939 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.949+0000 7f1b15c95640 4 set_mon_vals no callback set 2024-03-29T11:27:04.939 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.950+0000 7f1b15c95640 10 monclient: shutdown 2024-03-29T11:27:04.939 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.950+0000 7f1b15c95640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:04.939 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.950+0000 7f1b15c95640 1 -- 172.21.15.17:0/2710441017 >> v1:172.21.15.17:6790/0 conn(0x7f1b100b0770 legacy=0x7f1b10149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:04.939 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.951+0000 7f1b15c95640 1 -- 172.21.15.17:0/2710441017 shutdown_connections 2024-03-29T11:27:04.939 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.951+0000 7f1b15c95640 1 -- 172.21.15.17:0/2710441017 >> 172.21.15.17:0/2710441017 conn(0x7f1b1006b660 msgr2=0x7f1b1006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:04.940 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.951+0000 7f1b15c95640 1 -- 172.21.15.17:0/2710441017 shutdown_connections 2024-03-29T11:27:04.940 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.951+0000 7f1b15c95640 1 -- 172.21.15.17:0/2710441017 wait complete. 2024-03-29T11:27:04.940 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.952+0000 7f1b15c95640 10 monclient: build_initial_monmap 2024-03-29T11:27:04.940 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.952+0000 7f1b15c95640 10 monclient: monmap: 2024-03-29T11:27:04.940 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:27:04.940 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:04.940 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:27:04.941 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:27:04.941 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:04.941 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:04.941 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:27:04.941 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:27:04.941 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:27:04.941 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:04.941 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.953+0000 7f1b15c95640 1 librados: starting msgr at 2024-03-29T11:27:04.941 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.953+0000 7f1b15c95640 1 librados: starting objecter 2024-03-29T11:27:04.941 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.953+0000 7f1b15c95640 1 Processor -- start 2024-03-29T11:27:04.941 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.953+0000 7f1b15c95640 1 -- start start 2024-03-29T11:27:04.944 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.953+0000 7f1b15c95640 1 librados: setting wanted keys 2024-03-29T11:27:04.944 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.953+0000 7f1b15c95640 1 librados: calling monclient init 2024-03-29T11:27:04.945 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.953+0000 7f1b15c95640 10 monclient: init 2024-03-29T11:27:04.945 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.954+0000 7f1b15c95640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:04.945 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.954+0000 7f1b15c95640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:27:04.945 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.954+0000 7f1b15c95640 10 monclient(hunting): picked mon.noname-a con 0x7f1b100ac5a0 addr v1:172.21.15.17:6789/0 2024-03-29T11:27:04.945 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.954+0000 7f1b15c95640 10 monclient(hunting): picked mon.noname-c con 0x7f1b100b0770 addr v1:172.21.15.17:6790/0 2024-03-29T11:27:04.945 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.954+0000 7f1b15c95640 10 monclient(hunting): picked mon.noname-b con 0x7f1b1014a700 addr v1:172.21.15.94:6789/0 2024-03-29T11:27:04.945 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.954+0000 7f1b15c95640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b10127dc0 con 0x7f1b100ac5a0 2024-03-29T11:27:04.945 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.954+0000 7f1b15c95640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b10127950 con 0x7f1b1014a700 2024-03-29T11:27:04.945 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.954+0000 7f1b15c95640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b10127ae0 con 0x7f1b100b0770 2024-03-29T11:27:04.945 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.954+0000 7f1b15c95640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:04.945 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.954+0000 7f1b15c95640 10 monclient(hunting): authenticate will time out at 1331.159480s 2024-03-29T11:27:04.946 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.958+0000 7f1b0ffff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f1b1014a700 0x7f1b101de810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:47965/0 (socket says 172.21.15.17:47965) 2024-03-29T11:27:04.946 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.958+0000 7f1b0ffff640 1 -- 172.21.15.17:0/922790677 learned_addr learned my addr 172.21.15.17:0/922790677 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:27:04.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.959+0000 7f1af7fff640 10 client.?.objecter ms_handle_connect 0x7f1b1014a700 2024-03-29T11:27:04.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.959+0000 7f1af7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:04.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.959+0000 7f1af7fff640 10 client.?.objecter ms_handle_connect 0x7f1b100b0770 2024-03-29T11:27:04.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.959+0000 7f1af7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:04.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.959+0000 7f1af7fff640 10 client.?.objecter ms_handle_connect 0x7f1b100ac5a0 2024-03-29T11:27:04.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.959+0000 7f1af7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:04.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.959+0000 7f1af7fff640 1 -- 172.21.15.17:0/922790677 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3690861037 0 0) 0x7f1b10127950 con 0x7f1b1014a700 2024-03-29T11:27:04.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.959+0000 7f1af7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:04.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.959+0000 7f1af7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:04.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.959+0000 7f1af7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:04.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.959+0000 7f1af7fff640 10 monclient(hunting): my global_id is 4534 2024-03-29T11:27:04.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.959+0000 7f1af7fff640 1 -- 172.21.15.17:0/922790677 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1afc004eb0 con 0x7f1b1014a700 2024-03-29T11:27:04.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.959+0000 7f1af7fff640 1 -- 172.21.15.17:0/922790677 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2859864120 0 0) 0x7f1b10127ae0 con 0x7f1b100b0770 2024-03-29T11:27:04.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.959+0000 7f1af7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:04.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.959+0000 7f1af7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:04.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.959+0000 7f1af7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:04.948 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.959+0000 7f1af7fff640 10 monclient(hunting): my global_id is 4562 2024-03-29T11:27:04.948 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.959+0000 7f1af7fff640 1 -- 172.21.15.17:0/922790677 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b10127950 con 0x7f1b100b0770 2024-03-29T11:27:04.948 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.960+0000 7f1af7fff640 1 -- 172.21.15.17:0/922790677 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 169572828 0 0) 0x7f1b10127dc0 con 0x7f1b100ac5a0 2024-03-29T11:27:04.949 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.960+0000 7f1af7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:04.949 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.960+0000 7f1af7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:04.949 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.960+0000 7f1af7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:04.949 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.960+0000 7f1af7fff640 10 monclient(hunting): my global_id is 4506 2024-03-29T11:27:04.949 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.960+0000 7f1af7fff640 1 -- 172.21.15.17:0/922790677 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b10127ae0 con 0x7f1b100ac5a0 2024-03-29T11:27:04.949 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.960+0000 7f1af7fff640 1 -- 172.21.15.17:0/922790677 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 541118777 0 0) 0x7f1b10127950 con 0x7f1b100b0770 2024-03-29T11:27:04.949 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.960+0000 7f1af7fff640 1 -- 172.21.15.17:0/922790677 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b10127dc0 con 0x7f1b100b0770 2024-03-29T11:27:04.950 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.960+0000 7f1af7fff640 1 -- 172.21.15.17:0/922790677 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f1b08001b10 con 0x7f1b100b0770 2024-03-29T11:27:04.950 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.960+0000 7f1af7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:04.950 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.960+0000 7f1af7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:04.950 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.960+0000 7f1af7fff640 10 monclient(hunting): dump: 2024-03-29T11:27:04.950 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:27:04.951 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:27:04.951 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:04.951 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:04.951 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:04.951 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:04.951 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:27:04.951 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:27:04.951 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:27:04.951 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:04.951 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.960+0000 7f1af7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:04.951 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.960+0000 7f1af7fff640 1 -- 172.21.15.17:0/922790677 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1909436099 0 0) 0x7f1b10127ae0 con 0x7f1b100ac5a0 2024-03-29T11:27:04.951 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.960+0000 7f1af7fff640 1 -- 172.21.15.17:0/922790677 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b10127950 con 0x7f1b100ac5a0 2024-03-29T11:27:04.951 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.960+0000 7f1af7fff640 1 -- 172.21.15.17:0/922790677 <== mon.2 v1:172.21.15.17:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 100979444 0 0) 0x7f1b10127dc0 con 0x7f1b100b0770 2024-03-29T11:27:04.951 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.960+0000 7f1af7fff640 1 -- 172.21.15.17:0/922790677 >> v1:172.21.15.94:6789/0 conn(0x7f1b1014a700 legacy=0x7f1b101de810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:04.951 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.960+0000 7f1af7fff640 1 -- 172.21.15.17:0/922790677 >> v1:172.21.15.17:6789/0 conn(0x7f1b100ac5a0 legacy=0x7f1b10146ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:04.952 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.960+0000 7f1af7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:27:04.952 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.960+0000 7f1af7fff640 1 monclient: found mon.c 2024-03-29T11:27:04.952 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.961+0000 7f1af7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:04.952 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.961+0000 7f1af7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:27:04.952 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.961+0000 7f1af7fff640 1 -- 172.21.15.17:0/922790677 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b101471f0 con 0x7f1b100b0770 2024-03-29T11:27:04.952 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.961+0000 7f1af7fff640 10 monclient: _finish_auth 0 2024-03-29T11:27:04.952 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.961+0000 7f1af7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:27:04.952 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.961+0000 7f1af7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:04.952 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.961+0000 7f1b15c95640 5 monclient: authenticate success, global_id 4562 2024-03-29T11:27:04.952 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.961+0000 7f1b15c95640 10 monclient: _renew_subs 2024-03-29T11:27:04.952 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.961+0000 7f1b15c95640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:27:04.952 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.961+0000 7f1b15c95640 1 -- 172.21.15.17:0/922790677 --> v1:172.21.15.17:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b101473e0 con 0x7f1b100b0770 2024-03-29T11:27:04.952 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.961+0000 7f1b15c95640 10 client.4562.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:04.952 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.961+0000 7f1b15c95640 10 monclient: _renew_subs 2024-03-29T11:27:04.952 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.961+0000 7f1b15c95640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:27:04.952 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.961+0000 7f1b15c95640 1 -- 172.21.15.17:0/922790677 --> v1:172.21.15.17:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b10128c90 con 0x7f1b100b0770 2024-03-29T11:27:04.953 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.961+0000 7f1b15c95640 1 librados: init done 2024-03-29T11:27:04.953 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.962+0000 7f1b15c95640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:04.953 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.963+0000 7f1b15c95640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:04.953 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.963+0000 7f1b15c95640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:27:04.953 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.963+0000 7f1b15c95640 1 -- 172.21.15.17:0/922790677 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b100ab560 con 0x7f1b100b0770 2024-03-29T11:27:04.954 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.965+0000 7f1af7fff640 1 -- 172.21.15.17:0/922790677 <== mon.2 v1:172.21.15.17:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b08002060 con 0x7f1b100b0770 2024-03-29T11:27:04.954 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.965+0000 7f1af7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:04.954 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.965+0000 7f1af7fff640 1 -- 172.21.15.17:0/922790677 <== mon.2 v1:172.21.15.17:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f1b08003b10 con 0x7f1b100b0770 2024-03-29T11:27:04.954 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.965+0000 7f1af7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:04.954 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.965+0000 7f1af7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:04.954 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.965+0000 7f1af7fff640 10 monclient: dump: 2024-03-29T11:27:04.954 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:27:04.954 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:27:04.954 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:04.954 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:04.954 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:04.955 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:04.955 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:27:04.955 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:27:04.955 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:27:04.955 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:04.955 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.965+0000 7f1b0dffb640 4 set_mon_vals no callback set 2024-03-29T11:27:04.955 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.968+0000 7f1af7fff640 1 -- 172.21.15.17:0/922790677 <== mon.2 v1:172.21.15.17:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f1b08019ae0 con 0x7f1b100b0770 2024-03-29T11:27:04.956 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.969+0000 7f1b0f7fe640 10 monclient: get_auth_request con 0x7f1afc06dee0 auth_method 0 2024-03-29T11:27:04.956 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.969+0000 7f1af7fff640 1 -- 172.21.15.17:0/922790677 <== mon.2 v1:172.21.15.17:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f1b08083910 con 0x7f1b100b0770 2024-03-29T11:27:04.956 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.969+0000 7f1af7fff640 10 client.4562.objecter ms_dispatch 0x7f1b10000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:04.956 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.969+0000 7f1af7fff640 3 client.4562.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:04.956 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.969+0000 7f1af7fff640 3 client.4562.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:04.956 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.969+0000 7f1af7fff640 20 client.4562.objecter dump_active .. 0 homeless 2024-03-29T11:27:04.956 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.969+0000 7f1af7fff640 10 client.4562.objecter ms_handle_connect 0x7f1afc06dee0 2024-03-29T11:27:04.960 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.973+0000 7f1af7fff640 1 -- 172.21.15.17:0/922790677 <== mon.2 v1:172.21.15.17:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1b0804f1f0 con 0x7f1b100b0770 2024-03-29T11:27:04.960 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.973+0000 7f1af7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:04.961 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:04.973+0000 7f1af7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:05.293 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:05.305+0000 7f1b15c95640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:05.293 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:05.306+0000 7f1b15c95640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:05.293 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:05.306+0000 7f1b15c95640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:27:05.293 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:05.306+0000 7f1b15c95640 1 -- 172.21.15.17:0/922790677 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b10127f70 con 0x7f1b100b0770 2024-03-29T11:27:05.295 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:05.308+0000 7f1af7fff640 1 -- 172.21.15.17:0/922790677 <== mon.2 v1:172.21.15.17:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f1b10127f70 con 0x7f1b100b0770 2024-03-29T11:27:05.295 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:05.308+0000 7f1af7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:05.296 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:05.308+0000 7f1af7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:05.297 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:27:05.297 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:05.300 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:05.313+0000 7f1b15c95640 10 librados: watch_flush enter 2024-03-29T11:27:05.300 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:05.313+0000 7f1b15c95640 10 librados: watch_flush exit 2024-03-29T11:27:05.300 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:05.313+0000 7f1b15c95640 20 client.4562.objecter shutdown clearing up homeless session... 2024-03-29T11:27:05.300 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:05.313+0000 7f1b15c95640 10 client.4562.objecter successfully canceled tick 2024-03-29T11:27:05.301 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:05.313+0000 7f1b15c95640 1 -- 172.21.15.17:0/922790677 >> v1:172.21.15.17:6816/3236185052 conn(0x7f1afc06dee0 legacy=0x7f1afc0703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:05.301 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:05.313+0000 7f1b15c95640 10 monclient: shutdown 2024-03-29T11:27:05.301 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:05.313+0000 7f1b15c95640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:05.301 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:05.313+0000 7f1b15c95640 1 -- 172.21.15.17:0/922790677 >> v1:172.21.15.17:6790/0 conn(0x7f1b100b0770 legacy=0x7f1b101283a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:05.301 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:05.313+0000 7f1b15c95640 1 -- 172.21.15.17:0/922790677 shutdown_connections 2024-03-29T11:27:05.301 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:05.313+0000 7f1b15c95640 1 -- 172.21.15.17:0/922790677 >> 172.21.15.17:0/922790677 conn(0x7f1b1006b660 msgr2=0x7f1b1006f9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:05.302 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:05.315+0000 7f1b15c95640 1 -- 172.21.15.17:0/922790677 shutdown_connections 2024-03-29T11:27:05.302 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:05.315+0000 7f1b15c95640 1 -- 172.21.15.17:0/922790677 wait complete. 2024-03-29T11:27:05.302 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:05.315+0000 7f1b15c95640 1 librados: shutdown 2024-03-29T11:27:05.323 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:12.130 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:12.277 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.290+0000 7f0404b6d640 10 monclient: get_monmap_and_config 2024-03-29T11:27:12.278 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.290+0000 7f0404b6d640 10 monclient: build_initial_monmap 2024-03-29T11:27:12.278 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.290+0000 7f0404b6d640 10 monclient: monmap: 2024-03-29T11:27:12.278 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:27:12.278 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:27:12.278 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:27:12.291093+0000 2024-03-29T11:27:12.278 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:27:12.291093+0000 2024-03-29T11:27:12.278 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:12.278 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:12.278 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:27:12.278 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:27:12.278 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:27:12.278 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:12.281 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.294+0000 7f0404b6d640 1 Processor -- start 2024-03-29T11:27:12.283 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.294+0000 7f0404b6d640 1 -- start start 2024-03-29T11:27:12.283 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.294+0000 7f0404b6d640 10 monclient: init 2024-03-29T11:27:12.284 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.295+0000 7f0404b6d640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:12.284 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.295+0000 7f0404b6d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:27:12.284 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.295+0000 7f0404b6d640 10 monclient(hunting): picked mon.noname-c con 0x7f04001457f0 addr v1:172.21.15.94:6789/0 2024-03-29T11:27:12.284 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.295+0000 7f0404b6d640 10 monclient(hunting): picked mon.noname-b con 0x7f040013f650 addr v1:172.21.15.17:6790/0 2024-03-29T11:27:12.284 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.295+0000 7f0404b6d640 10 monclient(hunting): picked mon.noname-a con 0x7f04000a9d70 addr v1:172.21.15.17:6789/0 2024-03-29T11:27:12.284 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.295+0000 7f0404b6d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04000aaef0 con 0x7f04000a9d70 2024-03-29T11:27:12.285 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.295+0000 7f0404b6d640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0400145bd0 con 0x7f04001457f0 2024-03-29T11:27:12.285 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.295+0000 7f0404b6d640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0400145eb0 con 0x7f040013f650 2024-03-29T11:27:12.285 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.295+0000 7f0404b6d640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:12.285 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.295+0000 7f0404b6d640 10 monclient(hunting): authenticate will time out at 1338.500449s 2024-03-29T11:27:12.290 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.301+0000 7f03fdd9b640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f040013f650 0x7f04000b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:49263/0 (socket says 172.21.15.17:49263) 2024-03-29T11:27:12.290 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.301+0000 7f03fdd9b640 1 -- 172.21.15.17:0/2860953999 learned_addr learned my addr 172.21.15.17:0/2860953999 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:27:12.290 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.301+0000 7f03fd59a640 1 -- 172.21.15.17:0/2860953999 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2448251843 0 0) 0x7f0400145eb0 con 0x7f040013f650 2024-03-29T11:27:12.290 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.302+0000 7f03fd59a640 10 monclient(hunting): _negotiate 2024-03-29T11:27:12.290 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.302+0000 7f03fd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:12.290 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.302+0000 7f03fd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:12.290 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.302+0000 7f03fd59a640 10 monclient(hunting): my global_id is 4565 2024-03-29T11:27:12.291 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.302+0000 7f03fd59a640 1 -- 172.21.15.17:0/2860953999 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03e8003520 con 0x7f040013f650 2024-03-29T11:27:12.291 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.303+0000 7f03fd59a640 1 -- 172.21.15.17:0/2860953999 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 558966311 0 0) 0x7f03e8003520 con 0x7f040013f650 2024-03-29T11:27:12.291 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.303+0000 7f03fd59a640 1 -- 172.21.15.17:0/2860953999 >> v1:172.21.15.94:6789/0 conn(0x7f04001457f0 legacy=0x7f040013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:27:12.291 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.303+0000 7f03fd59a640 1 -- 172.21.15.17:0/2860953999 >> v1:172.21.15.17:6789/0 conn(0x7f04000a9d70 legacy=0x7f040014d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:12.291 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.303+0000 7f03fd59a640 10 monclient: _finish_hunting 0 2024-03-29T11:27:12.292 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.303+0000 7f03fd59a640 1 monclient: found mon.noname-b 2024-03-29T11:27:12.292 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.303+0000 7f03fd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:12.292 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.303+0000 7f03fd59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.17:6790/0 2024-03-29T11:27:12.292 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.303+0000 7f03fd59a640 1 -- 172.21.15.17:0/2860953999 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0400146340 con 0x7f040013f650 2024-03-29T11:27:12.292 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.303+0000 7f03fd59a640 10 monclient: _finish_auth 0 2024-03-29T11:27:12.292 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.303+0000 7f03fd59a640 10 monclient: _check_auth_tickets 2024-03-29T11:27:12.292 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.303+0000 7f03fd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:12.292 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.303+0000 7f03fd59a640 1 -- 172.21.15.17:0/2860953999 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f03f4001670 con 0x7f040013f650 2024-03-29T11:27:12.292 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.303+0000 7f03fd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:12.292 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.303+0000 7f03fd59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:12.292 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.303+0000 7f03fd59a640 10 monclient: dump: 2024-03-29T11:27:12.293 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:27:12.293 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:27:12.293 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:12.293 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:12.293 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:12.293 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:12.293 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:27:12.293 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:27:12.293 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:27:12.293 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:12.293 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.303+0000 7f0404b6d640 5 monclient: authenticate success, global_id 4565 2024-03-29T11:27:12.293 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.303+0000 7f0404b6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:12.294 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.304+0000 7f03fd59a640 1 -- 172.21.15.17:0/2860953999 <== mon.2 v1:172.21.15.17:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f03f4001840 con 0x7f040013f650 2024-03-29T11:27:12.294 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.304+0000 7f03fd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:12.294 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.304+0000 7f03fd59a640 1 -- 172.21.15.17:0/2860953999 <== mon.2 v1:172.21.15.17:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f03f4001c90 con 0x7f040013f650 2024-03-29T11:27:12.294 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.304+0000 7f03fd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:12.295 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.304+0000 7f03fd59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:12.295 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.304+0000 7f03fd59a640 10 monclient: dump: 2024-03-29T11:27:12.295 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:27:12.295 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:27:12.295 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:12.295 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:12.295 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:12.295 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:12.295 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:27:12.295 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:27:12.295 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:27:12.295 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:12.295 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.305+0000 7f0404b6d640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:12.295 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.305+0000 7f0404b6d640 4 set_mon_vals no callback set 2024-03-29T11:27:12.295 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.306+0000 7f0404b6d640 10 monclient: shutdown 2024-03-29T11:27:12.296 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.306+0000 7f0404b6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:12.296 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.306+0000 7f0404b6d640 1 -- 172.21.15.17:0/2860953999 >> v1:172.21.15.17:6790/0 conn(0x7f040013f650 legacy=0x7f04000b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:12.298 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.306+0000 7f0404b6d640 1 -- 172.21.15.17:0/2860953999 shutdown_connections 2024-03-29T11:27:12.299 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.306+0000 7f0404b6d640 1 -- 172.21.15.17:0/2860953999 >> 172.21.15.17:0/2860953999 conn(0x7f0400066a00 msgr2=0x7f0400066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:12.299 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.306+0000 7f0404b6d640 1 -- 172.21.15.17:0/2860953999 shutdown_connections 2024-03-29T11:27:12.299 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.306+0000 7f0404b6d640 1 -- 172.21.15.17:0/2860953999 wait complete. 2024-03-29T11:27:12.299 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.307+0000 7f0404b6d640 10 monclient: build_initial_monmap 2024-03-29T11:27:12.299 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.307+0000 7f0404b6d640 10 monclient: monmap: 2024-03-29T11:27:12.299 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:27:12.299 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:12.299 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:27:12.299 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:27:12.299 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:12.299 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:12.299 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:27:12.299 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:27:12.300 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:27:12.300 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:12.300 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.307+0000 7f0404b6d640 1 librados: starting msgr at 2024-03-29T11:27:12.300 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.307+0000 7f0404b6d640 1 librados: starting objecter 2024-03-29T11:27:12.300 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.308+0000 7f0404b6d640 1 Processor -- start 2024-03-29T11:27:12.300 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.308+0000 7f0404b6d640 1 -- start start 2024-03-29T11:27:12.300 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.308+0000 7f0404b6d640 1 librados: setting wanted keys 2024-03-29T11:27:12.300 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.308+0000 7f0404b6d640 1 librados: calling monclient init 2024-03-29T11:27:12.300 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.308+0000 7f0404b6d640 10 monclient: init 2024-03-29T11:27:12.300 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.308+0000 7f0404b6d640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:12.300 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.308+0000 7f0404b6d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:27:12.300 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.308+0000 7f0404b6d640 10 monclient(hunting): picked mon.noname-b con 0x7f04000a9d70 addr v1:172.21.15.94:6789/0 2024-03-29T11:27:12.300 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.308+0000 7f0404b6d640 10 monclient(hunting): picked mon.noname-a con 0x7f040013f650 addr v1:172.21.15.17:6789/0 2024-03-29T11:27:12.300 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.309+0000 7f0404b6d640 10 monclient(hunting): picked mon.noname-c con 0x7f04001457f0 addr v1:172.21.15.17:6790/0 2024-03-29T11:27:12.300 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.309+0000 7f0404b6d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f040013a7b0 con 0x7f040013f650 2024-03-29T11:27:12.301 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.309+0000 7f0404b6d640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f040013ac40 con 0x7f04000a9d70 2024-03-29T11:27:12.301 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.309+0000 7f0404b6d640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f040013a400 con 0x7f04001457f0 2024-03-29T11:27:12.301 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.309+0000 7f0404b6d640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:12.301 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.309+0000 7f0404b6d640 10 monclient(hunting): authenticate will time out at 1338.513920s 2024-03-29T11:27:12.307 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.314+0000 7f03fed9d640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f04001457f0 0x7f04001f34b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:49339/0 (socket says 172.21.15.17:49339) 2024-03-29T11:27:12.307 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.314+0000 7f03fed9d640 1 -- 172.21.15.17:0/4102394964 learned_addr learned my addr 172.21.15.17:0/4102394964 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:27:12.307 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.314+0000 7f03eeffd640 10 client.?.objecter ms_handle_connect 0x7f04001457f0 2024-03-29T11:27:12.308 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.314+0000 7f03eeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:12.308 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.314+0000 7f03eeffd640 1 -- 172.21.15.17:0/4102394964 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1580623079 0 0) 0x7f040013a400 con 0x7f04001457f0 2024-03-29T11:27:12.308 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.314+0000 7f03eeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:12.308 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.314+0000 7f03eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:12.308 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.314+0000 7f03eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:12.308 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.314+0000 7f03eeffd640 10 monclient(hunting): my global_id is 4568 2024-03-29T11:27:12.308 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.314+0000 7f03eeffd640 1 -- 172.21.15.17:0/4102394964 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03e0004eb0 con 0x7f04001457f0 2024-03-29T11:27:12.308 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.314+0000 7f03eeffd640 10 client.?.objecter ms_handle_connect 0x7f040013f650 2024-03-29T11:27:12.308 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.314+0000 7f03eeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:12.308 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.315+0000 7f03eeffd640 10 client.?.objecter ms_handle_connect 0x7f04000a9d70 2024-03-29T11:27:12.308 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.315+0000 7f03eeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:12.309 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.315+0000 7f03eeffd640 1 -- 172.21.15.17:0/4102394964 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3798532073 0 0) 0x7f03e0004eb0 con 0x7f04001457f0 2024-03-29T11:27:12.309 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.315+0000 7f03eeffd640 1 -- 172.21.15.17:0/4102394964 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f040013a400 con 0x7f04001457f0 2024-03-29T11:27:12.309 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.315+0000 7f03eeffd640 1 -- 172.21.15.17:0/4102394964 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f03f8001c60 con 0x7f04001457f0 2024-03-29T11:27:12.309 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.315+0000 7f03eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:12.309 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.315+0000 7f03eeffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:12.309 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.315+0000 7f03eeffd640 10 monclient(hunting): dump: 2024-03-29T11:27:12.309 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:27:12.309 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:27:12.309 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:12.309 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:12.309 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:12.309 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:12.309 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:27:12.309 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:27:12.310 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:27:12.310 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:12.310 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.315+0000 7f03eeffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:12.310 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.315+0000 7f03eeffd640 1 -- 172.21.15.17:0/4102394964 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3836033058 0 0) 0x7f040013ac40 con 0x7f04000a9d70 2024-03-29T11:27:12.310 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.315+0000 7f03eeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:12.310 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.315+0000 7f03eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:12.310 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.315+0000 7f03eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:12.310 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.315+0000 7f03eeffd640 10 monclient(hunting): my global_id is 4537 2024-03-29T11:27:12.310 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.315+0000 7f03eeffd640 1 -- 172.21.15.17:0/4102394964 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03e0004eb0 con 0x7f04000a9d70 2024-03-29T11:27:12.310 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.315+0000 7f03eeffd640 1 -- 172.21.15.17:0/4102394964 <== mon.2 v1:172.21.15.17:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2459691559 0 0) 0x7f040013a400 con 0x7f04001457f0 2024-03-29T11:27:12.311 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.315+0000 7f03eeffd640 1 -- 172.21.15.17:0/4102394964 >> v1:172.21.15.94:6789/0 conn(0x7f04000a9d70 legacy=0x7f04001395e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:12.311 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.315+0000 7f03eeffd640 1 -- 172.21.15.17:0/4102394964 >> v1:172.21.15.17:6789/0 conn(0x7f040013f650 legacy=0x7f0400139cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:12.311 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.315+0000 7f03eeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:27:12.311 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.315+0000 7f03eeffd640 1 monclient: found mon.c 2024-03-29T11:27:12.311 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.315+0000 7f03eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:12.311 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.315+0000 7f03eeffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:27:12.311 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.315+0000 7f03eeffd640 1 -- 172.21.15.17:0/4102394964 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f040013fda0 con 0x7f04001457f0 2024-03-29T11:27:12.311 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.315+0000 7f03eeffd640 10 monclient: _finish_auth 0 2024-03-29T11:27:12.311 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.315+0000 7f03eeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:27:12.311 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.315+0000 7f03eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:12.311 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.315+0000 7f0404b6d640 5 monclient: authenticate success, global_id 4568 2024-03-29T11:27:12.311 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.315+0000 7f0404b6d640 10 monclient: _renew_subs 2024-03-29T11:27:12.311 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.315+0000 7f0404b6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:27:12.311 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.315+0000 7f0404b6d640 1 -- 172.21.15.17:0/4102394964 --> v1:172.21.15.17:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f040008bd30 con 0x7f04001457f0 2024-03-29T11:27:12.311 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.315+0000 7f0404b6d640 10 client.4568.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:12.312 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.315+0000 7f0404b6d640 10 monclient: _renew_subs 2024-03-29T11:27:12.312 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.315+0000 7f0404b6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:27:12.312 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.315+0000 7f0404b6d640 1 -- 172.21.15.17:0/4102394964 --> v1:172.21.15.17:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0400141160 con 0x7f04001457f0 2024-03-29T11:27:12.312 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.315+0000 7f0404b6d640 1 librados: init done 2024-03-29T11:27:12.312 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.316+0000 7f03eeffd640 1 -- 172.21.15.17:0/4102394964 <== mon.2 v1:172.21.15.17:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f03f8001e40 con 0x7f04001457f0 2024-03-29T11:27:12.312 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.316+0000 7f03eeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:12.312 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.316+0000 7f03eeffd640 1 -- 172.21.15.17:0/4102394964 <== mon.2 v1:172.21.15.17:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f03f8003c20 con 0x7f04001457f0 2024-03-29T11:27:12.312 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.316+0000 7f03eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:12.312 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.316+0000 7f03eeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:12.312 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.316+0000 7f03eeffd640 10 monclient: dump: 2024-03-29T11:27:12.312 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:27:12.312 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:27:12.312 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:12.312 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:12.312 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:12.313 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:12.313 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:27:12.313 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:27:12.313 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:27:12.313 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:12.313 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.316+0000 7f03effff640 4 set_mon_vals no callback set 2024-03-29T11:27:12.313 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.317+0000 7f0404b6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:12.313 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.317+0000 7f0404b6d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:12.313 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.317+0000 7f0404b6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:27:12.313 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.317+0000 7f0404b6d640 1 -- 172.21.15.17:0/4102394964 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03e4005080 con 0x7f04001457f0 2024-03-29T11:27:12.314 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.322+0000 7f03eeffd640 1 -- 172.21.15.17:0/4102394964 <== mon.2 v1:172.21.15.17:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f03f8003e40 con 0x7f04001457f0 2024-03-29T11:27:12.314 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.322+0000 7f03eeffd640 1 -- 172.21.15.17:0/4102394964 <== mon.2 v1:172.21.15.17:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f03f8084d70 con 0x7f04001457f0 2024-03-29T11:27:12.314 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.322+0000 7f03eeffd640 10 client.4568.objecter ms_dispatch 0x7f0400000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:12.315 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.322+0000 7f03eeffd640 3 client.4568.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:12.315 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.322+0000 7f03eeffd640 3 client.4568.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:12.315 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.322+0000 7f03eeffd640 20 client.4568.objecter dump_active .. 0 homeless 2024-03-29T11:27:12.315 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.323+0000 7f03fe59c640 10 monclient: get_auth_request con 0x7f03e0076580 auth_method 0 2024-03-29T11:27:12.315 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.324+0000 7f03eeffd640 10 client.4568.objecter ms_handle_connect 0x7f03e0076580 2024-03-29T11:27:12.315 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.328+0000 7f03eeffd640 1 -- 172.21.15.17:0/4102394964 <== mon.2 v1:172.21.15.17:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f03f8050480 con 0x7f04001457f0 2024-03-29T11:27:12.315 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.328+0000 7f03eeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:12.316 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.328+0000 7f03eeffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:12.648 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.660+0000 7f0404b6d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:12.648 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.660+0000 7f0404b6d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:12.648 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.660+0000 7f0404b6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:27:12.648 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.660+0000 7f0404b6d640 1 -- 172.21.15.17:0/4102394964 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f03e4005330 con 0x7f04001457f0 2024-03-29T11:27:12.652 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.665+0000 7f03eeffd640 1 -- 172.21.15.17:0/4102394964 <== mon.2 v1:172.21.15.17:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f03f8052580 con 0x7f04001457f0 2024-03-29T11:27:12.652 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.665+0000 7f03eeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:12.652 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.665+0000 7f03eeffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:12.653 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:27:12.653 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:12.657 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.669+0000 7f0404b6d640 10 librados: watch_flush enter 2024-03-29T11:27:12.658 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.670+0000 7f0404b6d640 10 librados: watch_flush exit 2024-03-29T11:27:12.658 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.670+0000 7f0404b6d640 20 client.4568.objecter shutdown clearing up homeless session... 2024-03-29T11:27:12.658 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.670+0000 7f0404b6d640 10 client.4568.objecter successfully canceled tick 2024-03-29T11:27:12.658 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.671+0000 7f0404b6d640 1 -- 172.21.15.17:0/4102394964 >> v1:172.21.15.17:6816/3236185052 conn(0x7f03e0076580 legacy=0x7f03e0078a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:12.658 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.671+0000 7f0404b6d640 10 monclient: shutdown 2024-03-29T11:27:12.658 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.671+0000 7f0404b6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:12.658 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.671+0000 7f0404b6d640 1 -- 172.21.15.17:0/4102394964 >> v1:172.21.15.17:6790/0 conn(0x7f04001457f0 legacy=0x7f04001f34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:12.659 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.671+0000 7f0404b6d640 1 -- 172.21.15.17:0/4102394964 shutdown_connections 2024-03-29T11:27:12.659 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.671+0000 7f0404b6d640 1 -- 172.21.15.17:0/4102394964 >> 172.21.15.17:0/4102394964 conn(0x7f0400066a00 msgr2=0x7f04000b4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:12.659 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.672+0000 7f0404b6d640 1 -- 172.21.15.17:0/4102394964 shutdown_connections 2024-03-29T11:27:12.659 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.672+0000 7f0404b6d640 1 -- 172.21.15.17:0/4102394964 wait complete. 2024-03-29T11:27:12.660 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:12.672+0000 7f0404b6d640 1 librados: shutdown 2024-03-29T11:27:12.677 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:19.485 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:19.637 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.649+0000 7f720eb40640 10 monclient: get_monmap_and_config 2024-03-29T11:27:19.637 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.649+0000 7f720eb40640 10 monclient: build_initial_monmap 2024-03-29T11:27:19.637 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.649+0000 7f720eb40640 10 monclient: monmap: 2024-03-29T11:27:19.637 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:27:19.637 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:27:19.637 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:27:19.650430+0000 2024-03-29T11:27:19.638 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:27:19.650430+0000 2024-03-29T11:27:19.638 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:19.638 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:19.638 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:27:19.638 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:27:19.638 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:27:19.638 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:19.638 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.650+0000 7f720eb40640 1 Processor -- start 2024-03-29T11:27:19.639 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.650+0000 7f720eb40640 1 -- start start 2024-03-29T11:27:19.639 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.650+0000 7f720eb40640 10 monclient: init 2024-03-29T11:27:19.639 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.651+0000 7f720eb40640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:19.639 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.651+0000 7f720eb40640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:27:19.639 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.651+0000 7f720eb40640 10 monclient(hunting): picked mon.noname-c con 0x7f72081457f0 addr v1:172.21.15.94:6789/0 2024-03-29T11:27:19.639 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.651+0000 7f720eb40640 10 monclient(hunting): picked mon.noname-b con 0x7f720813f650 addr v1:172.21.15.17:6790/0 2024-03-29T11:27:19.639 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.651+0000 7f720eb40640 10 monclient(hunting): picked mon.noname-a con 0x7f72080a9d70 addr v1:172.21.15.17:6789/0 2024-03-29T11:27:19.639 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.651+0000 7f720eb40640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72080aaef0 con 0x7f72080a9d70 2024-03-29T11:27:19.640 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.651+0000 7f720eb40640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7208145bd0 con 0x7f72081457f0 2024-03-29T11:27:19.640 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.651+0000 7f720eb40640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7208145eb0 con 0x7f720813f650 2024-03-29T11:27:19.640 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.651+0000 7f720eb40640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:19.640 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.651+0000 7f720eb40640 10 monclient(hunting): authenticate will time out at 1345.856579s 2024-03-29T11:27:19.640 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.651+0000 7f7207fff640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f720813f650 0x7f72080b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:41240/0 (socket says 172.21.15.17:41240) 2024-03-29T11:27:19.640 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.651+0000 7f7207fff640 1 -- 172.21.15.17:0/1962575113 learned_addr learned my addr 172.21.15.17:0/1962575113 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:27:19.641 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.652+0000 7f72077fe640 1 -- 172.21.15.17:0/1962575113 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1841946718 0 0) 0x7f7208145eb0 con 0x7f720813f650 2024-03-29T11:27:19.642 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.652+0000 7f72077fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:19.642 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.652+0000 7f72077fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:19.642 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.652+0000 7f72077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:19.642 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.652+0000 7f72077fe640 10 monclient(hunting): my global_id is 4571 2024-03-29T11:27:19.642 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.652+0000 7f72077fe640 1 -- 172.21.15.17:0/1962575113 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71f8003520 con 0x7f720813f650 2024-03-29T11:27:19.642 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.652+0000 7f72077fe640 1 -- 172.21.15.17:0/1962575113 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3678756604 0 0) 0x7f72080aaef0 con 0x7f72080a9d70 2024-03-29T11:27:19.642 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.652+0000 7f72077fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:19.642 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.652+0000 7f72077fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:19.642 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.652+0000 7f72077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:19.642 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.652+0000 7f72077fe640 10 monclient(hunting): my global_id is 4512 2024-03-29T11:27:19.642 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.652+0000 7f72077fe640 1 -- 172.21.15.17:0/1962575113 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7208145eb0 con 0x7f72080a9d70 2024-03-29T11:27:19.642 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.653+0000 7f72077fe640 1 -- 172.21.15.17:0/1962575113 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1144311172 0 0) 0x7f7208145eb0 con 0x7f72080a9d70 2024-03-29T11:27:19.643 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.653+0000 7f72077fe640 1 -- 172.21.15.17:0/1962575113 >> v1:172.21.15.17:6790/0 conn(0x7f720813f650 legacy=0x7f72080b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:19.643 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.653+0000 7f72077fe640 1 -- 172.21.15.17:0/1962575113 >> v1:172.21.15.94:6789/0 conn(0x7f72081457f0 legacy=0x7f720813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:27:19.643 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.653+0000 7f72077fe640 10 monclient: _finish_hunting 0 2024-03-29T11:27:19.643 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.653+0000 7f72077fe640 1 monclient: found mon.noname-a 2024-03-29T11:27:19.643 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.653+0000 7f72077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:19.643 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.653+0000 7f72077fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.17:6789/0 2024-03-29T11:27:19.643 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.653+0000 7f72077fe640 1 -- 172.21.15.17:0/1962575113 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7208146340 con 0x7f72080a9d70 2024-03-29T11:27:19.643 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.653+0000 7f72077fe640 10 monclient: _finish_auth 0 2024-03-29T11:27:19.643 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.653+0000 7f72077fe640 10 monclient: _check_auth_tickets 2024-03-29T11:27:19.643 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.653+0000 7f72077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:19.643 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.653+0000 7f72077fe640 1 -- 172.21.15.17:0/1962575113 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f72000019f0 con 0x7f72080a9d70 2024-03-29T11:27:19.643 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.653+0000 7f72077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:19.643 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.653+0000 7f72077fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:19.644 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.653+0000 7f72077fe640 10 monclient: dump: 2024-03-29T11:27:19.644 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:27:19.644 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:27:19.644 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:19.644 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:19.644 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:19.644 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:19.644 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:27:19.644 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:27:19.644 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:27:19.644 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:19.644 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.653+0000 7f720eb40640 5 monclient: authenticate success, global_id 4512 2024-03-29T11:27:19.644 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.653+0000 7f720eb40640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:19.644 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.653+0000 7f72077fe640 1 -- 172.21.15.17:0/1962575113 <== mon.0 v1:172.21.15.17:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7200001b50 con 0x7f72080a9d70 2024-03-29T11:27:19.645 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.653+0000 7f72077fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:19.645 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.653+0000 7f72077fe640 1 -- 172.21.15.17:0/1962575113 <== mon.0 v1:172.21.15.17:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f7200001fa0 con 0x7f72080a9d70 2024-03-29T11:27:19.645 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.653+0000 7f72077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:19.645 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.653+0000 7f72077fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:19.645 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.653+0000 7f72077fe640 10 monclient: dump: 2024-03-29T11:27:19.645 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:27:19.645 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:27:19.645 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:19.645 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:19.645 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:19.645 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:19.645 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:27:19.646 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:27:19.646 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:27:19.646 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:19.646 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.653+0000 7f720eb40640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:19.646 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.653+0000 7f720eb40640 4 set_mon_vals no callback set 2024-03-29T11:27:19.651 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.654+0000 7f720eb40640 10 monclient: shutdown 2024-03-29T11:27:19.651 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.654+0000 7f720eb40640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:19.651 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.654+0000 7f720eb40640 1 -- 172.21.15.17:0/1962575113 >> v1:172.21.15.17:6789/0 conn(0x7f72080a9d70 legacy=0x7f720814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:19.651 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.654+0000 7f720eb40640 1 -- 172.21.15.17:0/1962575113 shutdown_connections 2024-03-29T11:27:19.652 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.655+0000 7f720eb40640 1 -- 172.21.15.17:0/1962575113 >> 172.21.15.17:0/1962575113 conn(0x7f7208066a00 msgr2=0x7f7208066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:19.652 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.655+0000 7f720eb40640 1 -- 172.21.15.17:0/1962575113 shutdown_connections 2024-03-29T11:27:19.652 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.655+0000 7f720eb40640 1 -- 172.21.15.17:0/1962575113 wait complete. 2024-03-29T11:27:19.652 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.656+0000 7f720eb40640 10 monclient: build_initial_monmap 2024-03-29T11:27:19.652 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.656+0000 7f720eb40640 10 monclient: monmap: 2024-03-29T11:27:19.652 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:27:19.652 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:19.652 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:27:19.652 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:27:19.652 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:19.652 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:19.652 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:27:19.652 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:27:19.652 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:27:19.652 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:19.653 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.656+0000 7f720eb40640 1 librados: starting msgr at 2024-03-29T11:27:19.653 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.656+0000 7f720eb40640 1 librados: starting objecter 2024-03-29T11:27:19.653 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.656+0000 7f720eb40640 1 Processor -- start 2024-03-29T11:27:19.653 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.656+0000 7f720eb40640 1 -- start start 2024-03-29T11:27:19.653 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.656+0000 7f720eb40640 1 librados: setting wanted keys 2024-03-29T11:27:19.653 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.656+0000 7f720eb40640 1 librados: calling monclient init 2024-03-29T11:27:19.653 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.656+0000 7f720eb40640 10 monclient: init 2024-03-29T11:27:19.653 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.657+0000 7f720eb40640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:19.653 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.657+0000 7f720eb40640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:27:19.653 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.657+0000 7f720eb40640 10 monclient(hunting): picked mon.noname-a con 0x7f72080a9d70 addr v1:172.21.15.17:6789/0 2024-03-29T11:27:19.653 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.657+0000 7f720eb40640 10 monclient(hunting): picked mon.noname-c con 0x7f720813f650 addr v1:172.21.15.17:6790/0 2024-03-29T11:27:19.653 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.657+0000 7f720eb40640 10 monclient(hunting): picked mon.noname-b con 0x7f72081457f0 addr v1:172.21.15.94:6789/0 2024-03-29T11:27:19.653 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.657+0000 7f720eb40640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72081387a0 con 0x7f72080a9d70 2024-03-29T11:27:19.653 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.658+0000 7f720eb40640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72080ad710 con 0x7f72081457f0 2024-03-29T11:27:19.654 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.658+0000 7f720eb40640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72081d81d0 con 0x7f720813f650 2024-03-29T11:27:19.654 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.658+0000 7f720eb40640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:19.654 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.658+0000 7f720eb40640 10 monclient(hunting): authenticate will time out at 1345.862987s 2024-03-29T11:27:19.654 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.658+0000 7f720c8dc640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f72080a9d70 0x7f72080ab600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:53742/0 (socket says 172.21.15.17:53742) 2024-03-29T11:27:19.654 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.658+0000 7f720c8dc640 1 -- 172.21.15.17:0/2842200578 learned_addr learned my addr 172.21.15.17:0/2842200578 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:27:19.654 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.658+0000 7f72057fa640 10 client.?.objecter ms_handle_connect 0x7f72080a9d70 2024-03-29T11:27:19.654 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.658+0000 7f72057fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:19.654 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.658+0000 7f72057fa640 10 client.?.objecter ms_handle_connect 0x7f720813f650 2024-03-29T11:27:19.654 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.658+0000 7f72057fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:19.654 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.658+0000 7f72057fa640 10 client.?.objecter ms_handle_connect 0x7f72081457f0 2024-03-29T11:27:19.655 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.658+0000 7f72057fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:19.655 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.658+0000 7f72057fa640 1 -- 172.21.15.17:0/2842200578 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2696218645 0 0) 0x7f72081387a0 con 0x7f72080a9d70 2024-03-29T11:27:19.655 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.658+0000 7f72057fa640 10 monclient(hunting): _negotiate 2024-03-29T11:27:19.655 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.658+0000 7f72057fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:19.655 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.658+0000 7f72057fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:19.655 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.658+0000 7f72057fa640 10 monclient(hunting): my global_id is 4515 2024-03-29T11:27:19.655 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.658+0000 7f72057fa640 1 -- 172.21.15.17:0/2842200578 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71ec004eb0 con 0x7f72080a9d70 2024-03-29T11:27:19.655 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.659+0000 7f72057fa640 1 -- 172.21.15.17:0/2842200578 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 169802546 0 0) 0x7f72080ad710 con 0x7f72081457f0 2024-03-29T11:27:19.655 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.659+0000 7f72057fa640 10 monclient(hunting): _negotiate 2024-03-29T11:27:19.655 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.659+0000 7f72057fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:19.655 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.659+0000 7f72057fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:19.655 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.659+0000 7f72057fa640 10 monclient(hunting): my global_id is 4540 2024-03-29T11:27:19.655 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.659+0000 7f72057fa640 1 -- 172.21.15.17:0/2842200578 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72081387a0 con 0x7f72081457f0 2024-03-29T11:27:19.656 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.659+0000 7f72057fa640 1 -- 172.21.15.17:0/2842200578 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3000638395 0 0) 0x7f71ec004eb0 con 0x7f72080a9d70 2024-03-29T11:27:19.656 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.659+0000 7f72057fa640 1 -- 172.21.15.17:0/2842200578 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72080ad710 con 0x7f72080a9d70 2024-03-29T11:27:19.656 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.659+0000 7f72057fa640 1 -- 172.21.15.17:0/2842200578 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f71fc0017d0 con 0x7f72080a9d70 2024-03-29T11:27:19.656 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.659+0000 7f72057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:19.656 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.659+0000 7f72057fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:19.656 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.659+0000 7f72057fa640 10 monclient(hunting): dump: 2024-03-29T11:27:19.656 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:27:19.656 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:27:19.656 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:19.656 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:19.656 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:19.656 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:19.656 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:27:19.656 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:27:19.656 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:27:19.656 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:19.657 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.659+0000 7f72057fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:19.657 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.659+0000 7f72057fa640 1 -- 172.21.15.17:0/2842200578 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3712979276 0 0) 0x7f72081d81d0 con 0x7f720813f650 2024-03-29T11:27:19.657 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.659+0000 7f72057fa640 10 monclient(hunting): _negotiate 2024-03-29T11:27:19.657 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.659+0000 7f72057fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:19.657 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.659+0000 7f72057fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:19.657 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.659+0000 7f72057fa640 10 monclient(hunting): my global_id is 4574 2024-03-29T11:27:19.657 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.659+0000 7f72057fa640 1 -- 172.21.15.17:0/2842200578 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71ec004eb0 con 0x7f720813f650 2024-03-29T11:27:19.657 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.659+0000 7f72057fa640 1 -- 172.21.15.17:0/2842200578 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1948509943 0 0) 0x7f72081387a0 con 0x7f72081457f0 2024-03-29T11:27:19.657 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.659+0000 7f72057fa640 1 -- 172.21.15.17:0/2842200578 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72081d81d0 con 0x7f72081457f0 2024-03-29T11:27:19.657 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.659+0000 7f72057fa640 1 -- 172.21.15.17:0/2842200578 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f7200001e60 con 0x7f72081457f0 2024-03-29T11:27:19.658 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.659+0000 7f72057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:19.658 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.659+0000 7f72057fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:19.658 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.659+0000 7f72057fa640 10 monclient(hunting): dump: 2024-03-29T11:27:19.658 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:27:19.658 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:27:19.658 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:19.658 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:19.658 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:19.658 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:19.658 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:27:19.658 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:27:19.658 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:27:19.658 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:19.659 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.659+0000 7f72057fa640 1 -- 172.21.15.17:0/2842200578 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 867567754 0 0) 0x7f71ec004eb0 con 0x7f720813f650 2024-03-29T11:27:19.659 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.659+0000 7f72057fa640 1 -- 172.21.15.17:0/2842200578 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72081387a0 con 0x7f720813f650 2024-03-29T11:27:19.659 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.659+0000 7f72057fa640 1 -- 172.21.15.17:0/2842200578 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f71f4001950 con 0x7f720813f650 2024-03-29T11:27:19.659 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.659+0000 7f72057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:19.659 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.659+0000 7f72057fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:19.659 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.659+0000 7f72057fa640 10 monclient(hunting): dump: 2024-03-29T11:27:19.659 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:27:19.659 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:27:19.659 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:19.659 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:19.659 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:19.659 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:19.659 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:27:19.659 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:27:19.660 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:27:19.660 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:19.660 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.659+0000 7f72057fa640 1 -- 172.21.15.17:0/2842200578 <== mon.1 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 898689462 0 0) 0x7f72081d81d0 con 0x7f72081457f0 2024-03-29T11:27:19.660 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.659+0000 7f72057fa640 1 -- 172.21.15.17:0/2842200578 >> v1:172.21.15.17:6790/0 conn(0x7f720813f650 legacy=0x7f7208137980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:19.660 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.659+0000 7f72057fa640 1 -- 172.21.15.17:0/2842200578 >> v1:172.21.15.17:6789/0 conn(0x7f72080a9d70 legacy=0x7f72080ab600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:19.660 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.660+0000 7f72057fa640 10 monclient: _finish_hunting 0 2024-03-29T11:27:19.660 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.660+0000 7f72057fa640 1 monclient: found mon.b 2024-03-29T11:27:19.660 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.660+0000 7f72057fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:19.660 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.660+0000 7f72057fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:27:19.660 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.660+0000 7f72057fa640 1 -- 172.21.15.17:0/2842200578 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72081d8360 con 0x7f72081457f0 2024-03-29T11:27:19.660 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.660+0000 7f72057fa640 10 monclient: _finish_auth 0 2024-03-29T11:27:19.660 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.660+0000 7f72057fa640 10 monclient: _check_auth_tickets 2024-03-29T11:27:19.660 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.660+0000 7f72057fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:19.660 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.660+0000 7f720eb40640 5 monclient: authenticate success, global_id 4540 2024-03-29T11:27:19.661 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.660+0000 7f720eb40640 10 monclient: _renew_subs 2024-03-29T11:27:19.661 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.660+0000 7f720eb40640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:27:19.661 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.660+0000 7f720eb40640 1 -- 172.21.15.17:0/2842200578 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72080abd10 con 0x7f72081457f0 2024-03-29T11:27:19.661 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.660+0000 7f720eb40640 10 client.4540.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:19.661 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.660+0000 7f720eb40640 10 monclient: _renew_subs 2024-03-29T11:27:19.661 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.660+0000 7f720eb40640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:27:19.661 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.660+0000 7f720eb40640 1 -- 172.21.15.17:0/2842200578 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f720813b950 con 0x7f72081457f0 2024-03-29T11:27:19.661 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.660+0000 7f720eb40640 1 librados: init done 2024-03-29T11:27:19.661 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.661+0000 7f72057fa640 1 -- 172.21.15.17:0/2842200578 <== mon.1 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7200001fc0 con 0x7f72081457f0 2024-03-29T11:27:19.661 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.661+0000 7f72057fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:19.661 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.661+0000 7f7206ffd640 4 set_mon_vals no callback set 2024-03-29T11:27:19.661 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.661+0000 7f720eb40640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:19.661 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.662+0000 7f72057fa640 1 -- 172.21.15.17:0/2842200578 <== mon.1 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f7200003dc0 con 0x7f72081457f0 2024-03-29T11:27:19.661 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.662+0000 7f720eb40640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:19.662 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.662+0000 7f720eb40640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:27:19.662 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.662+0000 7f720eb40640 1 -- 172.21.15.17:0/2842200578 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71e8005080 con 0x7f72081457f0 2024-03-29T11:27:19.662 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.662+0000 7f72057fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:19.662 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.662+0000 7f72057fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:19.662 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.662+0000 7f72057fa640 10 monclient: dump: 2024-03-29T11:27:19.662 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:27:19.662 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:27:19.662 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:19.662 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:19.662 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:19.662 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:19.662 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:27:19.662 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:27:19.662 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:27:19.663 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:19.663 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.663+0000 7f72057fa640 1 -- 172.21.15.17:0/2842200578 <== mon.1 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f7200019d90 con 0x7f72081457f0 2024-03-29T11:27:19.663 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.664+0000 7f720c8dc640 10 monclient: get_auth_request con 0x7f71ec06e0c0 auth_method 0 2024-03-29T11:27:19.663 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.664+0000 7f72057fa640 1 -- 172.21.15.17:0/2842200578 <== mon.1 v1:172.21.15.94:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f7200083b80 con 0x7f72081457f0 2024-03-29T11:27:19.664 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.664+0000 7f72057fa640 10 client.4540.objecter ms_dispatch 0x7f7208000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:19.664 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.664+0000 7f72057fa640 3 client.4540.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:19.664 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.664+0000 7f72057fa640 3 client.4540.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:19.664 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.664+0000 7f72057fa640 20 client.4540.objecter dump_active .. 0 homeless 2024-03-29T11:27:19.664 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.664+0000 7f72057fa640 10 client.4540.objecter ms_handle_connect 0x7f71ec06e0c0 2024-03-29T11:27:19.664 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.671+0000 7f72057fa640 1 -- 172.21.15.17:0/2842200578 <== mon.1 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f720004f460 con 0x7f72081457f0 2024-03-29T11:27:19.664 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.671+0000 7f72057fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:19.664 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:19.671+0000 7f72057fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:20.034 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:20.046+0000 7f720eb40640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:20.034 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:20.046+0000 7f720eb40640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:20.034 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:20.046+0000 7f720eb40640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:27:20.034 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:20.046+0000 7f720eb40640 1 -- 172.21.15.17:0/2842200578 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f71e8005330 con 0x7f72081457f0 2024-03-29T11:27:20.035 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:20.047+0000 7f72057fa640 1 -- 172.21.15.17:0/2842200578 <== mon.1 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f7200051390 con 0x7f72081457f0 2024-03-29T11:27:20.035 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:20.047+0000 7f72057fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:20.035 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:20.047+0000 7f72057fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:20.036 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:27:20.036 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:20.039 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:20.051+0000 7f720eb40640 10 librados: watch_flush enter 2024-03-29T11:27:20.040 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:20.051+0000 7f720eb40640 10 librados: watch_flush exit 2024-03-29T11:27:20.040 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:20.051+0000 7f720eb40640 20 client.4540.objecter shutdown clearing up homeless session... 2024-03-29T11:27:20.040 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:20.051+0000 7f720eb40640 10 client.4540.objecter successfully canceled tick 2024-03-29T11:27:20.040 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:20.051+0000 7f720eb40640 1 -- 172.21.15.17:0/2842200578 >> v1:172.21.15.17:6816/3236185052 conn(0x7f71ec06e0c0 legacy=0x7f71ec070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:20.040 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:20.051+0000 7f720eb40640 10 monclient: shutdown 2024-03-29T11:27:20.040 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:20.051+0000 7f720eb40640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:20.040 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:20.051+0000 7f720eb40640 1 -- 172.21.15.17:0/2842200578 >> v1:172.21.15.94:6789/0 conn(0x7f72081457f0 legacy=0x7f7208138090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:20.040 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:20.052+0000 7f720eb40640 1 -- 172.21.15.17:0/2842200578 shutdown_connections 2024-03-29T11:27:20.040 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:20.052+0000 7f720eb40640 1 -- 172.21.15.17:0/2842200578 >> 172.21.15.17:0/2842200578 conn(0x7f7208066a00 msgr2=0x7f720813cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:20.040 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:20.052+0000 7f720eb40640 1 -- 172.21.15.17:0/2842200578 shutdown_connections 2024-03-29T11:27:20.041 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:20.052+0000 7f720eb40640 1 -- 172.21.15.17:0/2842200578 wait complete. 2024-03-29T11:27:20.041 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:20.052+0000 7f720eb40640 1 librados: shutdown 2024-03-29T11:27:20.057 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:26.864 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:27.015 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.027+0000 7f9ff1d84640 10 monclient: get_monmap_and_config 2024-03-29T11:27:27.015 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.027+0000 7f9ff1d84640 10 monclient: build_initial_monmap 2024-03-29T11:27:27.016 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.027+0000 7f9ff1d84640 10 monclient: monmap: 2024-03-29T11:27:27.016 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:27:27.016 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:27:27.016 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:27:27.028489+0000 2024-03-29T11:27:27.016 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:27:27.028489+0000 2024-03-29T11:27:27.016 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:27.016 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:27.016 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:27:27.016 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:27:27.016 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:27:27.016 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:27.018 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.028+0000 7f9ff1d84640 1 Processor -- start 2024-03-29T11:27:27.018 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.028+0000 7f9ff1d84640 1 -- start start 2024-03-29T11:27:27.018 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.028+0000 7f9ff1d84640 10 monclient: init 2024-03-29T11:27:27.018 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.029+0000 7f9ff1d84640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:27.018 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.029+0000 7f9ff1d84640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:27:27.018 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.029+0000 7f9ff1d84640 10 monclient(hunting): picked mon.noname-a con 0x7f9fec0b46b0 addr v1:172.21.15.17:6789/0 2024-03-29T11:27:27.018 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.029+0000 7f9ff1d84640 10 monclient(hunting): picked mon.noname-c con 0x7f9fec0aeae0 addr v1:172.21.15.94:6789/0 2024-03-29T11:27:27.018 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.029+0000 7f9ff1d84640 10 monclient(hunting): picked mon.noname-b con 0x7f9fec13fa10 addr v1:172.21.15.17:6790/0 2024-03-29T11:27:27.018 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.029+0000 7f9ff1d84640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fec0b06a0 con 0x7f9fec0b46b0 2024-03-29T11:27:27.018 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.029+0000 7f9ff1d84640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fec0b4230 con 0x7f9fec0aeae0 2024-03-29T11:27:27.019 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.029+0000 7f9ff1d84640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fec0b0910 con 0x7f9fec13fa10 2024-03-29T11:27:27.019 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.029+0000 7f9ff1d84640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:27.019 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.029+0000 7f9ff1d84640 10 monclient(hunting): authenticate will time out at 1353.234582s 2024-03-29T11:27:27.024 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.036+0000 7f9fe3fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f9fec0aeae0 0x7f9fec13f300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:50273/0 (socket says 172.21.15.17:50273) 2024-03-29T11:27:27.024 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.036+0000 7f9fe3fff640 1 -- 172.21.15.17:0/376294618 learned_addr learned my addr 172.21.15.17:0/376294618 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:27:27.028 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.038+0000 7f9feaffd640 1 -- 172.21.15.17:0/376294618 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3452115496 0 0) 0x7f9fec0b0910 con 0x7f9fec13fa10 2024-03-29T11:27:27.028 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.038+0000 7f9feaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:27.028 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.038+0000 7f9feaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:27.028 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.038+0000 7f9feaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:27.028 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.038+0000 7f9feaffd640 10 monclient(hunting): my global_id is 4577 2024-03-29T11:27:27.028 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.038+0000 7f9feaffd640 1 -- 172.21.15.17:0/376294618 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fd8003520 con 0x7f9fec13fa10 2024-03-29T11:27:27.029 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.038+0000 7f9feaffd640 1 -- 172.21.15.17:0/376294618 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3969770688 0 0) 0x7f9fd8003520 con 0x7f9fec13fa10 2024-03-29T11:27:27.029 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.038+0000 7f9feaffd640 1 -- 172.21.15.17:0/376294618 >> v1:172.21.15.94:6789/0 conn(0x7f9fec0aeae0 legacy=0x7f9fec13f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:27.029 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.038+0000 7f9feaffd640 1 -- 172.21.15.17:0/376294618 >> v1:172.21.15.17:6789/0 conn(0x7f9fec0b46b0 legacy=0x7f9fec0b3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:27.029 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.038+0000 7f9feaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:27:27.029 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.039+0000 7f9feaffd640 1 monclient: found mon.noname-b 2024-03-29T11:27:27.029 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.039+0000 7f9feaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:27.029 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.039+0000 7f9feaffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.17:6790/0 2024-03-29T11:27:27.029 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.039+0000 7f9feaffd640 1 -- 172.21.15.17:0/376294618 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fec0af6d0 con 0x7f9fec13fa10 2024-03-29T11:27:27.029 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.039+0000 7f9feaffd640 10 monclient: _finish_auth 0 2024-03-29T11:27:27.029 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.039+0000 7f9feaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:27:27.029 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.039+0000 7f9feaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:27.029 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.039+0000 7f9feaffd640 1 -- 172.21.15.17:0/376294618 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f9fe4001a10 con 0x7f9fec13fa10 2024-03-29T11:27:27.029 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.039+0000 7f9feaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:27.029 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.039+0000 7f9feaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:27.029 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.039+0000 7f9feaffd640 10 monclient: dump: 2024-03-29T11:27:27.030 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:27:27.030 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:27:27.030 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:27.030 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:27.030 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:27.030 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:27.030 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:27:27.030 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:27:27.030 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:27:27.030 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:27.030 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.039+0000 7f9ff1d84640 5 monclient: authenticate success, global_id 4577 2024-03-29T11:27:27.030 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.039+0000 7f9ff1d84640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:27.030 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.040+0000 7f9feaffd640 1 -- 172.21.15.17:0/376294618 <== mon.2 v1:172.21.15.17:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9fe4001b70 con 0x7f9fec13fa10 2024-03-29T11:27:27.030 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.040+0000 7f9feaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:27.031 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.040+0000 7f9ff1d84640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:27.031 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.040+0000 7f9ff1d84640 4 set_mon_vals no callback set 2024-03-29T11:27:27.031 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.040+0000 7f9feaffd640 1 -- 172.21.15.17:0/376294618 <== mon.2 v1:172.21.15.17:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f9fe4001fc0 con 0x7f9fec13fa10 2024-03-29T11:27:27.031 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.040+0000 7f9feaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:27.031 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.040+0000 7f9feaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:27.031 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.040+0000 7f9feaffd640 10 monclient: dump: 2024-03-29T11:27:27.031 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:27:27.031 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:27:27.031 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:27.031 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:27.031 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:27.031 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:27.031 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:27:27.031 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:27:27.031 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:27:27.032 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:27.033 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.041+0000 7f9ff1d84640 10 monclient: shutdown 2024-03-29T11:27:27.033 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.041+0000 7f9ff1d84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:27.033 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.041+0000 7f9ff1d84640 1 -- 172.21.15.17:0/376294618 >> v1:172.21.15.17:6790/0 conn(0x7f9fec13fa10 legacy=0x7f9fec14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:27.034 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.041+0000 7f9ff1d84640 1 -- 172.21.15.17:0/376294618 shutdown_connections 2024-03-29T11:27:27.034 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.041+0000 7f9ff1d84640 1 -- 172.21.15.17:0/376294618 >> 172.21.15.17:0/376294618 conn(0x7f9fec066a00 msgr2=0x7f9fec066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:27.034 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.043+0000 7f9ff1d84640 1 -- 172.21.15.17:0/376294618 shutdown_connections 2024-03-29T11:27:27.034 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.044+0000 7f9ff1d84640 1 -- 172.21.15.17:0/376294618 wait complete. 2024-03-29T11:27:27.034 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.044+0000 7f9ff1d84640 10 monclient: build_initial_monmap 2024-03-29T11:27:27.034 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.044+0000 7f9ff1d84640 10 monclient: monmap: 2024-03-29T11:27:27.034 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:27:27.034 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:27.034 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:27:27.034 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:27:27.034 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:27.034 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:27.034 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:27:27.034 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:27:27.034 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:27:27.035 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:27.035 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.045+0000 7f9ff1d84640 1 librados: starting msgr at 2024-03-29T11:27:27.035 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.045+0000 7f9ff1d84640 1 librados: starting objecter 2024-03-29T11:27:27.035 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.045+0000 7f9ff1d84640 1 Processor -- start 2024-03-29T11:27:27.035 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.045+0000 7f9ff1d84640 1 -- start start 2024-03-29T11:27:27.035 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.045+0000 7f9ff1d84640 1 librados: setting wanted keys 2024-03-29T11:27:27.035 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.045+0000 7f9ff1d84640 1 librados: calling monclient init 2024-03-29T11:27:27.035 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.045+0000 7f9ff1d84640 10 monclient: init 2024-03-29T11:27:27.038 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.046+0000 7f9ff1d84640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:27.038 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.046+0000 7f9ff1d84640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:27:27.038 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.046+0000 7f9ff1d84640 10 monclient(hunting): picked mon.noname-b con 0x7f9fec0aeae0 addr v1:172.21.15.94:6789/0 2024-03-29T11:27:27.038 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.046+0000 7f9ff1d84640 10 monclient(hunting): picked mon.noname-a con 0x7f9fec0b46b0 addr v1:172.21.15.17:6789/0 2024-03-29T11:27:27.038 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.046+0000 7f9ff1d84640 10 monclient(hunting): picked mon.noname-c con 0x7f9fec13fa10 addr v1:172.21.15.17:6790/0 2024-03-29T11:27:27.038 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.046+0000 7f9ff1d84640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fec0ae580 con 0x7f9fec0b46b0 2024-03-29T11:27:27.038 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.046+0000 7f9ff1d84640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fec140e80 con 0x7f9fec0aeae0 2024-03-29T11:27:27.038 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.046+0000 7f9ff1d84640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fec1d7dc0 con 0x7f9fec13fa10 2024-03-29T11:27:27.038 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.046+0000 7f9ff1d84640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:27.038 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.046+0000 7f9ff1d84640 10 monclient(hunting): authenticate will time out at 1353.251153s 2024-03-29T11:27:27.039 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.049+0000 7f9feb7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f9fec0aeae0 0x7f9fec141b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:50369/0 (socket says 172.21.15.17:50369) 2024-03-29T11:27:27.039 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.049+0000 7f9feb7fe640 1 -- 172.21.15.17:0/1208896520 learned_addr learned my addr 172.21.15.17:0/1208896520 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:27:27.039 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.049+0000 7f9fe8ff9640 10 client.?.objecter ms_handle_connect 0x7f9fec0aeae0 2024-03-29T11:27:27.039 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.049+0000 7f9fe8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:27.039 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.050+0000 7f9fe8ff9640 1 -- 172.21.15.17:0/1208896520 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 107915500 0 0) 0x7f9fec140e80 con 0x7f9fec0aeae0 2024-03-29T11:27:27.039 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.050+0000 7f9fe8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:27:27.039 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.050+0000 7f9fe8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:27.039 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.050+0000 7f9fe8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:27.039 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.050+0000 7f9fe8ff9640 10 monclient(hunting): my global_id is 4546 2024-03-29T11:27:27.039 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.050+0000 7f9fe8ff9640 1 -- 172.21.15.17:0/1208896520 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fcc004eb0 con 0x7f9fec0aeae0 2024-03-29T11:27:27.040 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.050+0000 7f9fe8ff9640 1 -- 172.21.15.17:0/1208896520 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1708945768 0 0) 0x7f9fcc004eb0 con 0x7f9fec0aeae0 2024-03-29T11:27:27.040 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.050+0000 7f9fe8ff9640 1 -- 172.21.15.17:0/1208896520 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9fec140e80 con 0x7f9fec0aeae0 2024-03-29T11:27:27.040 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.051+0000 7f9fe8ff9640 1 -- 172.21.15.17:0/1208896520 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f9fdc001930 con 0x7f9fec0aeae0 2024-03-29T11:27:27.041 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.051+0000 7f9fe8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:27.041 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.051+0000 7f9fe8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:27.041 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.051+0000 7f9fe8ff9640 10 monclient(hunting): dump: 2024-03-29T11:27:27.041 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:27:27.041 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:27:27.041 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:27.041 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:27.041 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:27.041 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:27.041 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:27:27.041 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:27:27.041 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:27:27.041 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:27.041 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.051+0000 7f9fe8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:27.041 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.052+0000 7f9fe8ff9640 1 -- 172.21.15.17:0/1208896520 <== mon.1 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 866042262 0 0) 0x7f9fec140e80 con 0x7f9fec0aeae0 2024-03-29T11:27:27.042 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.052+0000 7f9fe8ff9640 1 -- 172.21.15.17:0/1208896520 >> v1:172.21.15.17:6790/0 conn(0x7f9fec13fa10 legacy=0x7f9fec136c30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:27:27.042 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.053+0000 7f9fe8ff9640 1 -- 172.21.15.17:0/1208896520 >> v1:172.21.15.17:6789/0 conn(0x7f9fec0b46b0 legacy=0x7f9fec1d76b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:27:27.042 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.055+0000 7f9fe8ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:27:27.045 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.057+0000 7f9fe8ff9640 1 monclient: found mon.b 2024-03-29T11:27:27.045 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.057+0000 7f9fe8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:27.045 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.057+0000 7f9fe8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:27:27.045 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.057+0000 7f9fe8ff9640 1 -- 172.21.15.17:0/1208896520 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fec0ada00 con 0x7f9fec0aeae0 2024-03-29T11:27:27.045 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.057+0000 7f9fe8ff9640 10 monclient: _finish_auth 0 2024-03-29T11:27:27.045 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.057+0000 7f9fe8ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:27:27.045 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.057+0000 7f9fe8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:27.047 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.057+0000 7f9ff1d84640 5 monclient: authenticate success, global_id 4546 2024-03-29T11:27:27.047 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.057+0000 7f9ff1d84640 10 monclient: _renew_subs 2024-03-29T11:27:27.047 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.057+0000 7f9ff1d84640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:27:27.047 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.057+0000 7f9ff1d84640 1 -- 172.21.15.17:0/1208896520 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fec0adc50 con 0x7f9fec0aeae0 2024-03-29T11:27:27.047 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.057+0000 7f9ff1d84640 10 client.4546.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:27.048 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.057+0000 7f9ff1d84640 10 monclient: _renew_subs 2024-03-29T11:27:27.048 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.057+0000 7f9ff1d84640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:27:27.048 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.057+0000 7f9ff1d84640 1 -- 172.21.15.17:0/1208896520 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fec141580 con 0x7f9fec0aeae0 2024-03-29T11:27:27.048 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.057+0000 7f9ff1d84640 1 librados: init done 2024-03-29T11:27:27.048 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.058+0000 7f9fe8ff9640 1 -- 172.21.15.17:0/1208896520 <== mon.1 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9fdc001d60 con 0x7f9fec0aeae0 2024-03-29T11:27:27.048 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.058+0000 7f9fe8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:27.048 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.058+0000 7f9fea7fc640 4 set_mon_vals no callback set 2024-03-29T11:27:27.048 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.059+0000 7f9fe8ff9640 1 -- 172.21.15.17:0/1208896520 <== mon.1 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f9fdc003a60 con 0x7f9fec0aeae0 2024-03-29T11:27:27.048 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.059+0000 7f9ff1d84640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:27.050 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.060+0000 7f9ff1d84640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:27.050 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.060+0000 7f9ff1d84640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:27:27.050 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.060+0000 7f9ff1d84640 1 -- 172.21.15.17:0/1208896520 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fd0005080 con 0x7f9fec0aeae0 2024-03-29T11:27:27.050 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.060+0000 7f9fe8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:27.050 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.060+0000 7f9fe8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:27.050 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.060+0000 7f9fe8ff9640 10 monclient: dump: 2024-03-29T11:27:27.050 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:27:27.050 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:27:27.050 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:27.050 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:27.050 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:27.051 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:27.051 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:27:27.051 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:27:27.051 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:27:27.051 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:27.051 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.060+0000 7f9fe8ff9640 1 -- 172.21.15.17:0/1208896520 <== mon.1 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f9fdc003bc0 con 0x7f9fec0aeae0 2024-03-29T11:27:27.051 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.061+0000 7f9fe8ff9640 1 -- 172.21.15.17:0/1208896520 <== mon.1 v1:172.21.15.94:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f9fdc084a90 con 0x7f9fec0aeae0 2024-03-29T11:27:27.051 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.061+0000 7f9fe8ff9640 10 client.4546.objecter ms_dispatch 0x7f9fec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:27.051 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.061+0000 7f9fe8ff9640 3 client.4546.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:27.051 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.061+0000 7f9fe8ff9640 3 client.4546.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:27.051 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.061+0000 7f9fe8ff9640 20 client.4546.objecter dump_active .. 0 homeless 2024-03-29T11:27:27.051 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.062+0000 7f9fe3fff640 10 monclient: get_auth_request con 0x7f9fcc06dbe0 auth_method 0 2024-03-29T11:27:27.051 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.062+0000 7f9fe8ff9640 10 client.4546.objecter ms_handle_connect 0x7f9fcc06dbe0 2024-03-29T11:27:27.054 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.066+0000 7f9fe8ff9640 1 -- 172.21.15.17:0/1208896520 <== mon.1 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9fdc050370 con 0x7f9fec0aeae0 2024-03-29T11:27:27.054 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.067+0000 7f9fe8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:27.054 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.067+0000 7f9fe8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:27.384 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.396+0000 7f9ff1d84640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:27.384 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.396+0000 7f9ff1d84640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:27.384 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.396+0000 7f9ff1d84640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:27:27.384 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.396+0000 7f9ff1d84640 1 -- 172.21.15.17:0/1208896520 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9fd0005330 con 0x7f9fec0aeae0 2024-03-29T11:27:27.385 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.397+0000 7f9fe8ff9640 1 -- 172.21.15.17:0/1208896520 <== mon.1 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f9fdc0522a0 con 0x7f9fec0aeae0 2024-03-29T11:27:27.385 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.397+0000 7f9fe8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:27.385 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.397+0000 7f9fe8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:27.388 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:27:27.388 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:27.391 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.403+0000 7f9ff1d84640 10 librados: watch_flush enter 2024-03-29T11:27:27.391 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.403+0000 7f9ff1d84640 10 librados: watch_flush exit 2024-03-29T11:27:27.391 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.403+0000 7f9ff1d84640 20 client.4546.objecter shutdown clearing up homeless session... 2024-03-29T11:27:27.391 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.403+0000 7f9ff1d84640 10 client.4546.objecter successfully canceled tick 2024-03-29T11:27:27.392 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.403+0000 7f9ff1d84640 1 -- 172.21.15.17:0/1208896520 >> v1:172.21.15.17:6816/3236185052 conn(0x7f9fcc06dbe0 legacy=0x7f9fcc0700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:27.392 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.403+0000 7f9ff1d84640 10 monclient: shutdown 2024-03-29T11:27:27.392 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.403+0000 7f9ff1d84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:27.392 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.403+0000 7f9ff1d84640 1 -- 172.21.15.17:0/1208896520 >> v1:172.21.15.94:6789/0 conn(0x7f9fec0aeae0 legacy=0x7f9fec141b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:27.392 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.403+0000 7f9ff1d84640 1 -- 172.21.15.17:0/1208896520 shutdown_connections 2024-03-29T11:27:27.392 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.403+0000 7f9ff1d84640 1 -- 172.21.15.17:0/1208896520 >> 172.21.15.17:0/1208896520 conn(0x7f9fec066a00 msgr2=0x7f9fec13f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:27.392 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.403+0000 7f9ff1d84640 1 -- 172.21.15.17:0/1208896520 shutdown_connections 2024-03-29T11:27:27.392 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.404+0000 7f9ff1d84640 1 -- 172.21.15.17:0/1208896520 wait complete. 2024-03-29T11:27:27.392 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:27.404+0000 7f9ff1d84640 1 librados: shutdown 2024-03-29T11:27:27.410 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:34.222 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:34.377 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.388+0000 7fdd9eab3640 10 monclient: get_monmap_and_config 2024-03-29T11:27:34.377 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.388+0000 7fdd9eab3640 10 monclient: build_initial_monmap 2024-03-29T11:27:34.377 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.388+0000 7fdd9eab3640 10 monclient: monmap: 2024-03-29T11:27:34.377 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:27:34.377 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:27:34.377 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:27:34.389940+0000 2024-03-29T11:27:34.377 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:27:34.389940+0000 2024-03-29T11:27:34.377 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:34.378 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:34.378 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:27:34.378 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:27:34.378 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:27:34.378 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:34.379 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.390+0000 7fdd9eab3640 1 Processor -- start 2024-03-29T11:27:34.379 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.390+0000 7fdd9eab3640 1 -- start start 2024-03-29T11:27:34.379 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.390+0000 7fdd9eab3640 10 monclient: init 2024-03-29T11:27:34.379 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.391+0000 7fdd9eab3640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:34.379 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.391+0000 7fdd9eab3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:27:34.382 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.391+0000 7fdd9eab3640 10 monclient(hunting): picked mon.noname-b con 0x7fdd981457f0 addr v1:172.21.15.17:6790/0 2024-03-29T11:27:34.382 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.393+0000 7fdd9eab3640 10 monclient(hunting): picked mon.noname-a con 0x7fdd9813f650 addr v1:172.21.15.17:6789/0 2024-03-29T11:27:34.382 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.393+0000 7fdd9eab3640 10 monclient(hunting): picked mon.noname-c con 0x7fdd980a9d70 addr v1:172.21.15.94:6789/0 2024-03-29T11:27:34.382 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.393+0000 7fdd9eab3640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd980aaef0 con 0x7fdd9813f650 2024-03-29T11:27:34.382 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.393+0000 7fdd9eab3640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd98145bd0 con 0x7fdd980a9d70 2024-03-29T11:27:34.382 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.393+0000 7fdd9eab3640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd98145eb0 con 0x7fdd981457f0 2024-03-29T11:27:34.382 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.393+0000 7fdd9eab3640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:34.382 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.393+0000 7fdd9eab3640 10 monclient(hunting): authenticate will time out at 1360.598643s 2024-03-29T11:27:34.383 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.394+0000 7fdd9c84f640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7fdd981457f0 0x7fdd9813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:39459/0 (socket says 172.21.15.17:39459) 2024-03-29T11:27:34.383 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.394+0000 7fdd9c84f640 1 -- 172.21.15.17:0/863017922 learned_addr learned my addr 172.21.15.17:0/863017922 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:27:34.383 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.395+0000 7fdd977fe640 1 -- 172.21.15.17:0/863017922 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1654812735 0 0) 0x7fdd98145eb0 con 0x7fdd981457f0 2024-03-29T11:27:34.383 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.395+0000 7fdd977fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:34.383 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.395+0000 7fdd977fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:34.383 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.395+0000 7fdd977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:34.384 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.395+0000 7fdd977fe640 10 monclient(hunting): my global_id is 4580 2024-03-29T11:27:34.384 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.395+0000 7fdd977fe640 1 -- 172.21.15.17:0/863017922 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd80003520 con 0x7fdd981457f0 2024-03-29T11:27:34.384 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.396+0000 7fdd977fe640 1 -- 172.21.15.17:0/863017922 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2065058213 0 0) 0x7fdd80003520 con 0x7fdd981457f0 2024-03-29T11:27:34.384 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.396+0000 7fdd977fe640 1 -- 172.21.15.17:0/863017922 >> v1:172.21.15.94:6789/0 conn(0x7fdd980a9d70 legacy=0x7fdd9814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:34.384 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.396+0000 7fdd977fe640 1 -- 172.21.15.17:0/863017922 >> v1:172.21.15.17:6789/0 conn(0x7fdd9813f650 legacy=0x7fdd980b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:34.387 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.398+0000 7fdd977fe640 10 monclient: _finish_hunting 0 2024-03-29T11:27:34.387 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.398+0000 7fdd977fe640 1 monclient: found mon.noname-b 2024-03-29T11:27:34.387 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.398+0000 7fdd977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:34.387 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.398+0000 7fdd977fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.17:6790/0 2024-03-29T11:27:34.387 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.398+0000 7fdd977fe640 1 -- 172.21.15.17:0/863017922 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd98146340 con 0x7fdd981457f0 2024-03-29T11:27:34.387 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.398+0000 7fdd977fe640 10 monclient: _finish_auth 0 2024-03-29T11:27:34.387 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.398+0000 7fdd977fe640 10 monclient: _check_auth_tickets 2024-03-29T11:27:34.387 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.398+0000 7fdd977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:34.387 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.398+0000 7fdd977fe640 1 -- 172.21.15.17:0/863017922 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fdd88001690 con 0x7fdd981457f0 2024-03-29T11:27:34.387 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.398+0000 7fdd977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:34.387 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.398+0000 7fdd977fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:34.387 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.398+0000 7fdd977fe640 10 monclient: dump: 2024-03-29T11:27:34.388 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:27:34.388 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:27:34.388 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:34.388 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:34.388 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:34.388 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:34.388 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:27:34.388 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:27:34.388 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:27:34.388 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:34.388 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.398+0000 7fdd977fe640 1 -- 172.21.15.17:0/863017922 <== mon.2 v1:172.21.15.17:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd88001920 con 0x7fdd981457f0 2024-03-29T11:27:34.388 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.398+0000 7fdd977fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:34.388 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.398+0000 7fdd977fe640 1 -- 172.21.15.17:0/863017922 <== mon.2 v1:172.21.15.17:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fdd88001d30 con 0x7fdd981457f0 2024-03-29T11:27:34.388 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.398+0000 7fdd977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:34.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.398+0000 7fdd977fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:34.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.398+0000 7fdd977fe640 10 monclient: dump: 2024-03-29T11:27:34.389 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:27:34.389 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:27:34.389 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:34.389 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:34.389 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:34.389 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:34.389 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:27:34.389 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:27:34.389 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:27:34.389 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:34.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.399+0000 7fdd9eab3640 5 monclient: authenticate success, global_id 4580 2024-03-29T11:27:34.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.399+0000 7fdd9eab3640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:34.390 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.399+0000 7fdd9eab3640 4 set_mon_vals no callback set 2024-03-29T11:27:34.390 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.400+0000 7fdd9eab3640 10 monclient: shutdown 2024-03-29T11:27:34.391 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.400+0000 7fdd9eab3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:34.391 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.400+0000 7fdd9eab3640 1 -- 172.21.15.17:0/863017922 >> v1:172.21.15.17:6790/0 conn(0x7fdd981457f0 legacy=0x7fdd9813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:34.391 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.400+0000 7fdd9eab3640 1 -- 172.21.15.17:0/863017922 shutdown_connections 2024-03-29T11:27:34.391 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.400+0000 7fdd9eab3640 1 -- 172.21.15.17:0/863017922 >> 172.21.15.17:0/863017922 conn(0x7fdd98066a00 msgr2=0x7fdd98066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:34.391 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.403+0000 7fdd9eab3640 1 -- 172.21.15.17:0/863017922 shutdown_connections 2024-03-29T11:27:34.392 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.403+0000 7fdd9eab3640 1 -- 172.21.15.17:0/863017922 wait complete. 2024-03-29T11:27:34.392 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.404+0000 7fdd9eab3640 10 monclient: build_initial_monmap 2024-03-29T11:27:34.392 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.404+0000 7fdd9eab3640 10 monclient: monmap: 2024-03-29T11:27:34.392 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:27:34.393 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:34.393 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:27:34.393 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:27:34.393 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:34.393 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:34.393 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:27:34.393 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:27:34.393 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:27:34.393 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:34.393 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.404+0000 7fdd9eab3640 1 librados: starting msgr at 2024-03-29T11:27:34.393 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.404+0000 7fdd9eab3640 1 librados: starting objecter 2024-03-29T11:27:34.393 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.404+0000 7fdd9eab3640 1 Processor -- start 2024-03-29T11:27:34.394 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.404+0000 7fdd9eab3640 1 -- start start 2024-03-29T11:27:34.394 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.404+0000 7fdd9eab3640 1 librados: setting wanted keys 2024-03-29T11:27:34.394 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.404+0000 7fdd9eab3640 1 librados: calling monclient init 2024-03-29T11:27:34.394 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.404+0000 7fdd9eab3640 10 monclient: init 2024-03-29T11:27:34.395 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.405+0000 7fdd9eab3640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:34.396 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.405+0000 7fdd9eab3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:27:34.396 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.406+0000 7fdd9eab3640 10 monclient(hunting): picked mon.noname-b con 0x7fdd980a9d70 addr v1:172.21.15.94:6789/0 2024-03-29T11:27:34.396 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.406+0000 7fdd9eab3640 10 monclient(hunting): picked mon.noname-a con 0x7fdd9813f650 addr v1:172.21.15.17:6789/0 2024-03-29T11:27:34.402 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.406+0000 7fdd9eab3640 10 monclient(hunting): picked mon.noname-c con 0x7fdd981457f0 addr v1:172.21.15.17:6790/0 2024-03-29T11:27:34.402 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.406+0000 7fdd9c84f640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fdd980a9d70 0x7fdd9813cd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:48082/0 (socket says 172.21.15.17:48082) 2024-03-29T11:27:34.402 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.408+0000 7fdd9c84f640 1 -- 172.21.15.17:0/2555567535 learned_addr learned my addr 172.21.15.17:0/2555567535 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:27:34.402 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.408+0000 7fdd9eab3640 1 -- 172.21.15.17:0/2555567535 --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd98139330 con 0x7fdd9813f650 2024-03-29T11:27:34.402 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.408+0000 7fdd957fa640 10 client.?.objecter ms_handle_connect 0x7fdd980a9d70 2024-03-29T11:27:34.402 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.408+0000 7fdd957fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:34.402 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.408+0000 7fdd9eab3640 1 -- 172.21.15.17:0/2555567535 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd98139610 con 0x7fdd980a9d70 2024-03-29T11:27:34.402 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.408+0000 7fdd9eab3640 1 -- 172.21.15.17:0/2555567535 --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd9813ad20 con 0x7fdd981457f0 2024-03-29T11:27:34.402 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.408+0000 7fdd9eab3640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:34.403 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.408+0000 7fdd9eab3640 10 monclient(hunting): authenticate will time out at 1360.613582s 2024-03-29T11:27:34.403 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.409+0000 7fdd957fa640 10 client.?.objecter ms_handle_connect 0x7fdd981457f0 2024-03-29T11:27:34.403 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.409+0000 7fdd957fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:34.403 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.409+0000 7fdd957fa640 1 -- 172.21.15.17:0/2555567535 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 726632847 0 0) 0x7fdd98139610 con 0x7fdd980a9d70 2024-03-29T11:27:34.403 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.409+0000 7fdd957fa640 10 monclient(hunting): _negotiate 2024-03-29T11:27:34.403 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.409+0000 7fdd957fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:34.403 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.409+0000 7fdd957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:34.403 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.409+0000 7fdd957fa640 10 monclient(hunting): my global_id is 4552 2024-03-29T11:27:34.403 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.409+0000 7fdd957fa640 1 -- 172.21.15.17:0/2555567535 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd84004eb0 con 0x7fdd980a9d70 2024-03-29T11:27:34.403 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.409+0000 7fdd957fa640 1 -- 172.21.15.17:0/2555567535 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3874748223 0 0) 0x7fdd9813ad20 con 0x7fdd981457f0 2024-03-29T11:27:34.403 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.409+0000 7fdd957fa640 10 monclient(hunting): _negotiate 2024-03-29T11:27:34.403 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.409+0000 7fdd957fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:34.403 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.409+0000 7fdd957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:34.403 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.409+0000 7fdd957fa640 10 monclient(hunting): my global_id is 4583 2024-03-29T11:27:34.404 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.409+0000 7fdd957fa640 1 -- 172.21.15.17:0/2555567535 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd98139610 con 0x7fdd981457f0 2024-03-29T11:27:34.404 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.409+0000 7fdd957fa640 1 -- 172.21.15.17:0/2555567535 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 953622269 0 0) 0x7fdd84004eb0 con 0x7fdd980a9d70 2024-03-29T11:27:34.404 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.409+0000 7fdd957fa640 1 -- 172.21.15.17:0/2555567535 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd84005490 con 0x7fdd980a9d70 2024-03-29T11:27:34.404 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.409+0000 7fdd957fa640 1 -- 172.21.15.17:0/2555567535 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fdd88001d30 con 0x7fdd980a9d70 2024-03-29T11:27:34.404 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.409+0000 7fdd957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:34.404 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.409+0000 7fdd957fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:34.404 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.409+0000 7fdd957fa640 10 monclient(hunting): dump: 2024-03-29T11:27:34.404 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:27:34.404 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:27:34.404 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:34.404 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:34.404 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:34.404 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:34.404 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:27:34.405 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:27:34.405 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:27:34.405 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:34.405 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.410+0000 7fdd957fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:34.405 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.410+0000 7fdd957fa640 1 -- 172.21.15.17:0/2555567535 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3354709434 0 0) 0x7fdd98139610 con 0x7fdd981457f0 2024-03-29T11:27:34.405 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.410+0000 7fdd957fa640 1 -- 172.21.15.17:0/2555567535 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd84004eb0 con 0x7fdd981457f0 2024-03-29T11:27:34.405 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.410+0000 7fdd957fa640 1 -- 172.21.15.17:0/2555567535 <== mon.1 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 878737430 0 0) 0x7fdd84005490 con 0x7fdd980a9d70 2024-03-29T11:27:34.405 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.410+0000 7fdd957fa640 1 -- 172.21.15.17:0/2555567535 >> v1:172.21.15.17:6790/0 conn(0x7fdd981457f0 legacy=0x7fdd981ee8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:34.405 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.410+0000 7fdd957fa640 1 -- 172.21.15.17:0/2555567535 >> v1:172.21.15.17:6789/0 conn(0x7fdd9813f650 legacy=0x7fdd9813d440 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:27:34.405 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.411+0000 7fdd957fa640 10 monclient: _finish_hunting 0 2024-03-29T11:27:34.405 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.411+0000 7fdd957fa640 1 monclient: found mon.b 2024-03-29T11:27:34.405 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.411+0000 7fdd957fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:34.405 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.411+0000 7fdd957fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:27:34.405 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.411+0000 7fdd957fa640 1 -- 172.21.15.17:0/2555567535 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd9813aeb0 con 0x7fdd980a9d70 2024-03-29T11:27:34.405 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.411+0000 7fdd957fa640 10 monclient: _finish_auth 0 2024-03-29T11:27:34.406 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.411+0000 7fdd957fa640 10 monclient: _check_auth_tickets 2024-03-29T11:27:34.406 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.411+0000 7fdd957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:34.406 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.411+0000 7fdd9eab3640 5 monclient: authenticate success, global_id 4552 2024-03-29T11:27:34.406 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.411+0000 7fdd9eab3640 10 monclient: _renew_subs 2024-03-29T11:27:34.406 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.411+0000 7fdd9eab3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:27:34.406 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.411+0000 7fdd9eab3640 1 -- 172.21.15.17:0/2555567535 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd98143cd0 con 0x7fdd980a9d70 2024-03-29T11:27:34.406 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.411+0000 7fdd9eab3640 10 client.4552.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:34.406 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.411+0000 7fdd9eab3640 10 monclient: _renew_subs 2024-03-29T11:27:34.406 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.411+0000 7fdd9eab3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:27:34.406 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.411+0000 7fdd9eab3640 1 -- 172.21.15.17:0/2555567535 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd98144aa0 con 0x7fdd980a9d70 2024-03-29T11:27:34.406 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.411+0000 7fdd9eab3640 1 librados: init done 2024-03-29T11:27:34.406 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.413+0000 7fdd9eab3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:34.406 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.413+0000 7fdd957fa640 1 -- 172.21.15.17:0/2555567535 <== mon.1 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd88004a90 con 0x7fdd980a9d70 2024-03-29T11:27:34.407 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.413+0000 7fdd9eab3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:34.407 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.413+0000 7fdd9eab3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:27:34.407 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.413+0000 7fdd9eab3640 1 -- 172.21.15.17:0/2555567535 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd98075f10 con 0x7fdd980a9d70 2024-03-29T11:27:34.407 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.413+0000 7fdd957fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:34.408 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.413+0000 7fdd957fa640 1 -- 172.21.15.17:0/2555567535 <== mon.1 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fdd88004ee0 con 0x7fdd980a9d70 2024-03-29T11:27:34.408 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.413+0000 7fdd957fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:34.408 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.413+0000 7fdd957fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:34.408 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.413+0000 7fdd967fc640 4 set_mon_vals no callback set 2024-03-29T11:27:34.408 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.413+0000 7fdd957fa640 10 monclient: dump: 2024-03-29T11:27:34.409 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:27:34.409 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:27:34.409 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:34.409 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:34.409 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:34.409 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:34.409 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:27:34.409 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:27:34.409 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:27:34.409 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:34.409 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.414+0000 7fdd957fa640 1 -- 172.21.15.17:0/2555567535 <== mon.1 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7fdd88005100 con 0x7fdd980a9d70 2024-03-29T11:27:34.409 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.415+0000 7fdd957fa640 1 -- 172.21.15.17:0/2555567535 <== mon.1 v1:172.21.15.94:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7fdd88084bb0 con 0x7fdd980a9d70 2024-03-29T11:27:34.409 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.415+0000 7fdd957fa640 10 client.4552.objecter ms_dispatch 0x7fdd98000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:34.409 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.415+0000 7fdd957fa640 3 client.4552.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:34.410 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.415+0000 7fdd957fa640 3 client.4552.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:34.410 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.415+0000 7fdd957fa640 20 client.4552.objecter dump_active .. 0 homeless 2024-03-29T11:27:34.410 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.415+0000 7fdd97fff640 10 monclient: get_auth_request con 0x7fdd8406ded0 auth_method 0 2024-03-29T11:27:34.410 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.415+0000 7fdd957fa640 10 client.4552.objecter ms_handle_connect 0x7fdd8406ded0 2024-03-29T11:27:34.412 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.424+0000 7fdd957fa640 1 -- 172.21.15.17:0/2555567535 <== mon.1 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdd88050490 con 0x7fdd980a9d70 2024-03-29T11:27:34.412 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.424+0000 7fdd957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:34.412 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.424+0000 7fdd957fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:34.738 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.750+0000 7fdd9eab3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:34.739 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.750+0000 7fdd9eab3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:34.739 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.750+0000 7fdd9eab3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:27:34.739 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.750+0000 7fdd9eab3640 1 -- 172.21.15.17:0/2555567535 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd98072080 con 0x7fdd980a9d70 2024-03-29T11:27:34.739 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.751+0000 7fdd957fa640 1 -- 172.21.15.17:0/2555567535 <== mon.1 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7fdd880523c0 con 0x7fdd980a9d70 2024-03-29T11:27:34.739 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.751+0000 7fdd957fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:34.739 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.751+0000 7fdd957fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:34.740 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:27:34.740 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:34.743 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.754+0000 7fdd9eab3640 10 librados: watch_flush enter 2024-03-29T11:27:34.743 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.754+0000 7fdd9eab3640 10 librados: watch_flush exit 2024-03-29T11:27:34.743 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.755+0000 7fdd9eab3640 20 client.4552.objecter shutdown clearing up homeless session... 2024-03-29T11:27:34.743 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.755+0000 7fdd9eab3640 10 client.4552.objecter successfully canceled tick 2024-03-29T11:27:34.743 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.755+0000 7fdd9eab3640 1 -- 172.21.15.17:0/2555567535 >> v1:172.21.15.17:6816/3236185052 conn(0x7fdd8406ded0 legacy=0x7fdd84070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:34.744 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.755+0000 7fdd9eab3640 10 monclient: shutdown 2024-03-29T11:27:34.744 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.755+0000 7fdd9eab3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:34.744 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.755+0000 7fdd9eab3640 1 -- 172.21.15.17:0/2555567535 >> v1:172.21.15.94:6789/0 conn(0x7fdd980a9d70 legacy=0x7fdd9813cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:34.744 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.755+0000 7fdd9eab3640 1 -- 172.21.15.17:0/2555567535 shutdown_connections 2024-03-29T11:27:34.744 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.755+0000 7fdd9eab3640 1 -- 172.21.15.17:0/2555567535 >> 172.21.15.17:0/2555567535 conn(0x7fdd98066a00 msgr2=0x7fdd980b2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:34.744 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.755+0000 7fdd9eab3640 1 -- 172.21.15.17:0/2555567535 shutdown_connections 2024-03-29T11:27:34.744 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.755+0000 7fdd9eab3640 1 -- 172.21.15.17:0/2555567535 wait complete. 2024-03-29T11:27:34.744 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:34.756+0000 7fdd9eab3640 1 librados: shutdown 2024-03-29T11:27:34.760 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:41.568 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:41.710 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.720+0000 7fd8d32a8640 10 monclient: get_monmap_and_config 2024-03-29T11:27:41.710 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.720+0000 7fd8d32a8640 10 monclient: build_initial_monmap 2024-03-29T11:27:41.710 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.720+0000 7fd8d32a8640 10 monclient: monmap: 2024-03-29T11:27:41.710 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:27:41.710 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:27:41.710 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:27:41.722452+0000 2024-03-29T11:27:41.710 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:27:41.722452+0000 2024-03-29T11:27:41.710 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:41.710 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:41.710 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:27:41.710 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:27:41.711 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:27:41.711 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:41.712 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.721+0000 7fd8d32a8640 1 Processor -- start 2024-03-29T11:27:41.712 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.721+0000 7fd8d32a8640 1 -- start start 2024-03-29T11:27:41.713 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.721+0000 7fd8d32a8640 10 monclient: init 2024-03-29T11:27:41.713 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.722+0000 7fd8d32a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:41.713 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.722+0000 7fd8d32a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:27:41.713 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.722+0000 7fd8d32a8640 10 monclient(hunting): picked mon.noname-a con 0x7fd8cc0b4790 addr v1:172.21.15.17:6789/0 2024-03-29T11:27:41.713 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.722+0000 7fd8d32a8640 10 monclient(hunting): picked mon.noname-c con 0x7fd8cc0aea90 addr v1:172.21.15.94:6789/0 2024-03-29T11:27:41.713 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.722+0000 7fd8d32a8640 10 monclient(hunting): picked mon.noname-b con 0x7fd8cc0b39f0 addr v1:172.21.15.17:6790/0 2024-03-29T11:27:41.713 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.722+0000 7fd8d32a8640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8cc0b0c60 con 0x7fd8cc0b4790 2024-03-29T11:27:41.713 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.722+0000 7fd8d32a8640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8cc0af020 con 0x7fd8cc0aea90 2024-03-29T11:27:41.713 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.722+0000 7fd8d32a8640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8cc13fbd0 con 0x7fd8cc0b39f0 2024-03-29T11:27:41.713 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.722+0000 7fd8d32a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:41.713 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.722+0000 7fd8d32a8640 10 monclient(hunting): authenticate will time out at 1367.928494s 2024-03-29T11:27:41.713 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.723+0000 7fd8d1044640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fd8cc0b4790 0x7fd8cc0b1080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:37568/0 (socket says 172.21.15.17:37568) 2024-03-29T11:27:41.713 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.723+0000 7fd8d1044640 1 -- 172.21.15.17:0/956187594 learned_addr learned my addr 172.21.15.17:0/956187594 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:27:41.715 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.724+0000 7fd8c3fff640 1 -- 172.21.15.17:0/956187594 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2191691234 0 0) 0x7fd8cc0b0c60 con 0x7fd8cc0b4790 2024-03-29T11:27:41.715 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.724+0000 7fd8c3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:41.715 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.724+0000 7fd8c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:41.715 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.724+0000 7fd8c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:41.715 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.724+0000 7fd8c3fff640 10 monclient(hunting): my global_id is 4521 2024-03-29T11:27:41.715 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.724+0000 7fd8c3fff640 1 -- 172.21.15.17:0/956187594 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8b8003520 con 0x7fd8cc0b4790 2024-03-29T11:27:41.715 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.724+0000 7fd8c3fff640 1 -- 172.21.15.17:0/956187594 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3546700718 0 0) 0x7fd8b8003520 con 0x7fd8cc0b4790 2024-03-29T11:27:41.715 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.724+0000 7fd8c3fff640 1 -- 172.21.15.17:0/956187594 >> v1:172.21.15.17:6790/0 conn(0x7fd8cc0b39f0 legacy=0x7fd8cc13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:41.716 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.724+0000 7fd8c3fff640 1 -- 172.21.15.17:0/956187594 >> v1:172.21.15.94:6789/0 conn(0x7fd8cc0aea90 legacy=0x7fd8cc0af540 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:41.717 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.726+0000 7fd8c3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:27:41.717 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.726+0000 7fd8c3fff640 1 monclient: found mon.noname-a 2024-03-29T11:27:41.717 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.726+0000 7fd8c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:41.717 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.726+0000 7fd8c3fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.17:6789/0 2024-03-29T11:27:41.717 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.726+0000 7fd8c3fff640 1 -- 172.21.15.17:0/956187594 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8cc13fd60 con 0x7fd8cc0b4790 2024-03-29T11:27:41.717 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.726+0000 7fd8c3fff640 10 monclient: _finish_auth 0 2024-03-29T11:27:41.717 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.726+0000 7fd8c3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:27:41.717 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.726+0000 7fd8c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:41.717 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.726+0000 7fd8c3fff640 1 -- 172.21.15.17:0/956187594 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fd8bc001690 con 0x7fd8cc0b4790 2024-03-29T11:27:41.718 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.726+0000 7fd8c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:41.718 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.726+0000 7fd8c3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:41.718 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.726+0000 7fd8c3fff640 10 monclient: dump: 2024-03-29T11:27:41.718 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:27:41.718 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:27:41.718 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:41.718 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:41.718 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:41.718 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:41.718 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:27:41.718 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:27:41.718 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:27:41.718 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:41.718 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.726+0000 7fd8d32a8640 5 monclient: authenticate success, global_id 4521 2024-03-29T11:27:41.719 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.726+0000 7fd8d32a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:41.719 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.726+0000 7fd8c3fff640 1 -- 172.21.15.17:0/956187594 <== mon.0 v1:172.21.15.17:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8bc001920 con 0x7fd8cc0b4790 2024-03-29T11:27:41.719 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.726+0000 7fd8c3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:41.719 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.726+0000 7fd8d32a8640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:41.719 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.726+0000 7fd8d32a8640 4 set_mon_vals no callback set 2024-03-29T11:27:41.719 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.726+0000 7fd8c3fff640 1 -- 172.21.15.17:0/956187594 <== mon.0 v1:172.21.15.17:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fd8bc001d30 con 0x7fd8cc0b4790 2024-03-29T11:27:41.719 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.726+0000 7fd8c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:41.719 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.726+0000 7fd8c3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:41.719 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.726+0000 7fd8c3fff640 10 monclient: dump: 2024-03-29T11:27:41.719 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:27:41.719 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:27:41.719 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:41.719 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:41.719 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:41.719 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:41.720 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:27:41.720 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:27:41.720 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:27:41.720 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:41.720 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.727+0000 7fd8d32a8640 10 monclient: shutdown 2024-03-29T11:27:41.720 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.727+0000 7fd8d32a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:41.720 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.727+0000 7fd8d32a8640 1 -- 172.21.15.17:0/956187594 >> v1:172.21.15.17:6789/0 conn(0x7fd8cc0b4790 legacy=0x7fd8cc0b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:41.720 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.727+0000 7fd8d32a8640 1 -- 172.21.15.17:0/956187594 shutdown_connections 2024-03-29T11:27:41.720 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.727+0000 7fd8d32a8640 1 -- 172.21.15.17:0/956187594 >> 172.21.15.17:0/956187594 conn(0x7fd8cc066a00 msgr2=0x7fd8cc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:41.727 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.728+0000 7fd8d32a8640 1 -- 172.21.15.17:0/956187594 shutdown_connections 2024-03-29T11:27:41.727 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.728+0000 7fd8d32a8640 1 -- 172.21.15.17:0/956187594 wait complete. 2024-03-29T11:27:41.727 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.729+0000 7fd8d32a8640 10 monclient: build_initial_monmap 2024-03-29T11:27:41.727 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.729+0000 7fd8d32a8640 10 monclient: monmap: 2024-03-29T11:27:41.727 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:27:41.727 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:41.728 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:27:41.728 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:27:41.728 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:41.728 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:41.728 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:27:41.728 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:27:41.728 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:27:41.728 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:41.728 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.729+0000 7fd8d32a8640 1 librados: starting msgr at 2024-03-29T11:27:41.728 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.729+0000 7fd8d32a8640 1 librados: starting objecter 2024-03-29T11:27:41.728 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.729+0000 7fd8d32a8640 1 Processor -- start 2024-03-29T11:27:41.728 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.730+0000 7fd8d32a8640 1 -- start start 2024-03-29T11:27:41.728 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.730+0000 7fd8d32a8640 1 librados: setting wanted keys 2024-03-29T11:27:41.728 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.730+0000 7fd8d32a8640 1 librados: calling monclient init 2024-03-29T11:27:41.729 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.730+0000 7fd8d32a8640 10 monclient: init 2024-03-29T11:27:41.729 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.730+0000 7fd8d32a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:41.729 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.730+0000 7fd8d32a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:27:41.729 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.730+0000 7fd8d32a8640 10 monclient(hunting): picked mon.noname-a con 0x7fd8cc0b39f0 addr v1:172.21.15.17:6789/0 2024-03-29T11:27:41.729 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.730+0000 7fd8d32a8640 10 monclient(hunting): picked mon.noname-b con 0x7fd8cc0ae310 addr v1:172.21.15.94:6789/0 2024-03-29T11:27:41.729 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.730+0000 7fd8d32a8640 10 monclient(hunting): picked mon.noname-c con 0x7fd8cc135f00 addr v1:172.21.15.17:6790/0 2024-03-29T11:27:41.729 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.730+0000 7fd8d32a8640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8cc1dbcb0 con 0x7fd8cc0b39f0 2024-03-29T11:27:41.729 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.730+0000 7fd8d32a8640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8cc136fa0 con 0x7fd8cc0ae310 2024-03-29T11:27:41.729 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.730+0000 7fd8d32a8640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8cc137280 con 0x7fd8cc135f00 2024-03-29T11:27:41.729 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.730+0000 7fd8d32a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:41.729 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.730+0000 7fd8d32a8640 10 monclient(hunting): authenticate will time out at 1367.936726s 2024-03-29T11:27:41.729 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.731+0000 7fd8d1845640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7fd8cc135f00 0x7fd8cc1db5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:45326/0 (socket says 172.21.15.17:45326) 2024-03-29T11:27:41.729 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.731+0000 7fd8d1845640 1 -- 172.21.15.17:0/998127019 learned_addr learned my addr 172.21.15.17:0/998127019 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:27:41.729 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.731+0000 7fd8c1ffb640 10 client.?.objecter ms_handle_connect 0x7fd8cc135f00 2024-03-29T11:27:41.730 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.731+0000 7fd8c1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:41.730 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.731+0000 7fd8c1ffb640 1 -- 172.21.15.17:0/998127019 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1560908488 0 0) 0x7fd8cc137280 con 0x7fd8cc135f00 2024-03-29T11:27:41.730 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.731+0000 7fd8c1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:41.730 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.731+0000 7fd8c1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:41.730 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.731+0000 7fd8c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:41.730 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.731+0000 7fd8c1ffb640 10 monclient(hunting): my global_id is 4586 2024-03-29T11:27:41.730 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.731+0000 7fd8c1ffb640 1 -- 172.21.15.17:0/998127019 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8b0004eb0 con 0x7fd8cc135f00 2024-03-29T11:27:41.730 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.731+0000 7fd8c1ffb640 1 -- 172.21.15.17:0/998127019 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 589888655 0 0) 0x7fd8b0004eb0 con 0x7fd8cc135f00 2024-03-29T11:27:41.730 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.732+0000 7fd8c1ffb640 1 -- 172.21.15.17:0/998127019 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8cc137280 con 0x7fd8cc135f00 2024-03-29T11:27:41.730 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.732+0000 7fd8c1ffb640 1 -- 172.21.15.17:0/998127019 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fd8c4001bc0 con 0x7fd8cc135f00 2024-03-29T11:27:41.730 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.732+0000 7fd8c1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:41.730 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.732+0000 7fd8c1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:41.730 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.732+0000 7fd8c1ffb640 10 monclient(hunting): dump: 2024-03-29T11:27:41.731 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:27:41.731 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:27:41.731 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:41.731 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:41.731 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:41.731 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:41.731 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:27:41.731 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:27:41.731 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:27:41.731 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:41.731 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.732+0000 7fd8c1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:41.731 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.732+0000 7fd8c1ffb640 1 -- 172.21.15.17:0/998127019 <== mon.2 v1:172.21.15.17:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1216888179 0 0) 0x7fd8cc137280 con 0x7fd8cc135f00 2024-03-29T11:27:41.731 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.732+0000 7fd8c1ffb640 1 -- 172.21.15.17:0/998127019 >> v1:172.21.15.94:6789/0 conn(0x7fd8cc0ae310 legacy=0x7fd8cc1dae90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:27:41.731 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.734+0000 7fd8c1ffb640 1 -- 172.21.15.17:0/998127019 >> v1:172.21.15.17:6789/0 conn(0x7fd8cc0b39f0 legacy=0x7fd8cc137980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:41.731 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.734+0000 7fd8c1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:27:41.731 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.734+0000 7fd8c1ffb640 1 monclient: found mon.c 2024-03-29T11:27:41.732 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.734+0000 7fd8c1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:41.732 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.734+0000 7fd8c1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:27:41.732 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.734+0000 7fd8c1ffb640 1 -- 172.21.15.17:0/998127019 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8cc137560 con 0x7fd8cc135f00 2024-03-29T11:27:41.732 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.734+0000 7fd8c1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:27:41.732 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.734+0000 7fd8c1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:27:41.732 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.734+0000 7fd8c1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:41.732 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.734+0000 7fd8c1ffb640 10 client.?.objecter ms_handle_connect 0x7fd8cc0b39f0 2024-03-29T11:27:41.732 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.734+0000 7fd8c1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:41.732 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.734+0000 7fd8d32a8640 5 monclient: authenticate success, global_id 4586 2024-03-29T11:27:41.732 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.734+0000 7fd8d32a8640 10 monclient: _renew_subs 2024-03-29T11:27:41.732 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.734+0000 7fd8d32a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:27:41.732 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.734+0000 7fd8d32a8640 1 -- 172.21.15.17:0/998127019 --> v1:172.21.15.17:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8cc0adb10 con 0x7fd8cc135f00 2024-03-29T11:27:41.732 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.734+0000 7fd8c1ffb640 1 -- 172.21.15.17:0/998127019 <== mon.2 v1:172.21.15.17:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8c4001d20 con 0x7fd8cc135f00 2024-03-29T11:27:41.732 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.734+0000 7fd8c1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:41.732 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.734+0000 7fd8c1ffb640 1 -- 172.21.15.17:0/998127019 <== mon.2 v1:172.21.15.17:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fd8c4003bb0 con 0x7fd8cc135f00 2024-03-29T11:27:41.733 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.734+0000 7fd8c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:41.733 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.734+0000 7fd8c1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:41.733 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.734+0000 7fd8c1ffb640 10 monclient: dump: 2024-03-29T11:27:41.733 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:27:41.733 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:27:41.733 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:41.733 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:41.733 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:41.733 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:41.733 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:27:41.733 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:27:41.733 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:27:41.733 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:41.733 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.734+0000 7fd8d32a8640 10 client.4586.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:41.733 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.734+0000 7fd8d32a8640 10 monclient: _renew_subs 2024-03-29T11:27:41.734 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.734+0000 7fd8d32a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:27:41.734 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.734+0000 7fd8d32a8640 1 -- 172.21.15.17:0/998127019 --> v1:172.21.15.17:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8cc0ac280 con 0x7fd8cc135f00 2024-03-29T11:27:41.734 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.734+0000 7fd8d32a8640 1 librados: init done 2024-03-29T11:27:41.734 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.734+0000 7fd8c37fe640 4 set_mon_vals no callback set 2024-03-29T11:27:41.734 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.735+0000 7fd8d32a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:41.734 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.736+0000 7fd8d32a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:41.734 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.736+0000 7fd8d32a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:27:41.734 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.736+0000 7fd8d32a8640 1 -- 172.21.15.17:0/998127019 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8cc075f10 con 0x7fd8cc135f00 2024-03-29T11:27:41.734 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.737+0000 7fd8c1ffb640 1 -- 172.21.15.17:0/998127019 <== mon.2 v1:172.21.15.17:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7fd8c4003d10 con 0x7fd8cc135f00 2024-03-29T11:27:41.734 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.737+0000 7fd8d1044640 10 monclient: get_auth_request con 0x7fd8b006dbc0 auth_method 0 2024-03-29T11:27:41.734 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.737+0000 7fd8c1ffb640 1 -- 172.21.15.17:0/998127019 <== mon.2 v1:172.21.15.17:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7fd8c4054bc0 con 0x7fd8cc135f00 2024-03-29T11:27:41.734 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.737+0000 7fd8c1ffb640 10 client.4586.objecter ms_dispatch 0x7fd8cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:41.734 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.737+0000 7fd8c1ffb640 3 client.4586.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:41.734 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.737+0000 7fd8c1ffb640 3 client.4586.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:41.735 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.737+0000 7fd8c1ffb640 20 client.4586.objecter dump_active .. 0 homeless 2024-03-29T11:27:41.735 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.738+0000 7fd8c1ffb640 10 client.4586.objecter ms_handle_connect 0x7fd8b006dbc0 2024-03-29T11:27:41.735 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.745+0000 7fd8c1ffb640 1 -- 172.21.15.17:0/998127019 <== mon.2 v1:172.21.15.17:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd8c4050500 con 0x7fd8cc135f00 2024-03-29T11:27:41.735 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.745+0000 7fd8c1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:41.735 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:41.745+0000 7fd8c1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:42.060 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:42.071+0000 7fd8d32a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:42.061 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:42.071+0000 7fd8d32a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:42.061 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:42.071+0000 7fd8d32a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:27:42.061 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:42.071+0000 7fd8d32a8640 1 -- 172.21.15.17:0/998127019 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd8cc0b5da0 con 0x7fd8cc135f00 2024-03-29T11:27:42.061 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:42.072+0000 7fd8c1ffb640 1 -- 172.21.15.17:0/998127019 <== mon.2 v1:172.21.15.17:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7fd8c4052600 con 0x7fd8cc135f00 2024-03-29T11:27:42.061 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:42.072+0000 7fd8c1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:42.061 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:42.072+0000 7fd8c1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:42.062 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:27:42.062 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:42.065 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:42.075+0000 7fd8d32a8640 10 librados: watch_flush enter 2024-03-29T11:27:42.065 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:42.075+0000 7fd8d32a8640 10 librados: watch_flush exit 2024-03-29T11:27:42.065 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:42.075+0000 7fd8d32a8640 20 client.4586.objecter shutdown clearing up homeless session... 2024-03-29T11:27:42.065 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:42.075+0000 7fd8d32a8640 10 client.4586.objecter successfully canceled tick 2024-03-29T11:27:42.065 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:42.075+0000 7fd8d32a8640 1 -- 172.21.15.17:0/998127019 >> v1:172.21.15.17:6816/3236185052 conn(0x7fd8b006dbc0 legacy=0x7fd8b0070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:42.065 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:42.075+0000 7fd8d32a8640 10 monclient: shutdown 2024-03-29T11:27:42.065 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:42.075+0000 7fd8d32a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:42.065 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:42.075+0000 7fd8d32a8640 1 -- 172.21.15.17:0/998127019 >> v1:172.21.15.17:6790/0 conn(0x7fd8cc135f00 legacy=0x7fd8cc1db5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:42.066 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:42.076+0000 7fd8d32a8640 1 -- 172.21.15.17:0/998127019 shutdown_connections 2024-03-29T11:27:42.066 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:42.076+0000 7fd8d32a8640 1 -- 172.21.15.17:0/998127019 >> 172.21.15.17:0/998127019 conn(0x7fd8cc066a00 msgr2=0x7fd8cc0b1080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:42.066 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:42.076+0000 7fd8d32a8640 1 -- 172.21.15.17:0/998127019 shutdown_connections 2024-03-29T11:27:42.066 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:42.076+0000 7fd8d32a8640 1 -- 172.21.15.17:0/998127019 wait complete. 2024-03-29T11:27:42.066 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:42.076+0000 7fd8d32a8640 1 librados: shutdown 2024-03-29T11:27:42.080 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:48.887 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:49.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.034+0000 7fbc16c4e640 10 monclient: get_monmap_and_config 2024-03-29T11:27:49.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.034+0000 7fbc16c4e640 10 monclient: build_initial_monmap 2024-03-29T11:27:49.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.035+0000 7fbc16c4e640 10 monclient: monmap: 2024-03-29T11:27:49.026 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:27:49.026 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:27:49.026 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:27:49.035992+0000 2024-03-29T11:27:49.026 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:27:49.035992+0000 2024-03-29T11:27:49.026 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:49.026 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:49.026 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:27:49.027 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:27:49.027 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:27:49.027 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:49.027 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.036+0000 7fbc16c4e640 1 Processor -- start 2024-03-29T11:27:49.027 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.036+0000 7fbc16c4e640 1 -- start start 2024-03-29T11:27:49.027 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.036+0000 7fbc16c4e640 10 monclient: init 2024-03-29T11:27:49.027 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.036+0000 7fbc16c4e640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:49.027 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.036+0000 7fbc16c4e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:27:49.027 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.037+0000 7fbc16c4e640 10 monclient(hunting): picked mon.noname-b con 0x7fbc100ae250 addr v1:172.21.15.17:6790/0 2024-03-29T11:27:49.027 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.037+0000 7fbc16c4e640 10 monclient(hunting): picked mon.noname-a con 0x7fbc100a9720 addr v1:172.21.15.17:6789/0 2024-03-29T11:27:49.027 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.037+0000 7fbc16c4e640 10 monclient(hunting): picked mon.noname-c con 0x7fbc1014bba0 addr v1:172.21.15.94:6789/0 2024-03-29T11:27:49.027 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.037+0000 7fbc16c4e640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc100a9b80 con 0x7fbc100a9720 2024-03-29T11:27:49.028 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.037+0000 7fbc16c4e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc100aa7b0 con 0x7fbc1014bba0 2024-03-29T11:27:49.028 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.037+0000 7fbc16c4e640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc100afbf0 con 0x7fbc100ae250 2024-03-29T11:27:49.028 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.037+0000 7fbc16c4e640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:49.028 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.037+0000 7fbc16c4e640 10 monclient(hunting): authenticate will time out at 1375.241978s 2024-03-29T11:27:49.028 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.037+0000 7fbc149ea640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7fbc100ae250 0x7fbc10142b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:56478/0 (socket says 172.21.15.17:56478) 2024-03-29T11:27:49.028 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.037+0000 7fbc149ea640 1 -- 172.21.15.17:0/2327107565 learned_addr learned my addr 172.21.15.17:0/2327107565 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:27:49.032 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.037+0000 7fbc0f7fe640 1 -- 172.21.15.17:0/2327107565 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 723109117 0 0) 0x7fbc100aa7b0 con 0x7fbc1014bba0 2024-03-29T11:27:49.032 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.037+0000 7fbc0f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:49.032 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.037+0000 7fbc0f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:49.032 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.037+0000 7fbc0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:49.032 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.037+0000 7fbc0f7fe640 10 monclient(hunting): my global_id is 4555 2024-03-29T11:27:49.032 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.037+0000 7fbc0f7fe640 1 -- 172.21.15.17:0/2327107565 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc00003520 con 0x7fbc1014bba0 2024-03-29T11:27:49.032 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.037+0000 7fbc0f7fe640 1 -- 172.21.15.17:0/2327107565 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3516624256 0 0) 0x7fbc100afbf0 con 0x7fbc100ae250 2024-03-29T11:27:49.032 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.037+0000 7fbc0f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:49.032 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.037+0000 7fbc0f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:49.032 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.037+0000 7fbc0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:49.032 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.037+0000 7fbc0f7fe640 10 monclient(hunting): my global_id is 4589 2024-03-29T11:27:49.033 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.037+0000 7fbc0f7fe640 1 -- 172.21.15.17:0/2327107565 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc100aa7b0 con 0x7fbc100ae250 2024-03-29T11:27:49.033 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.038+0000 7fbc0f7fe640 1 -- 172.21.15.17:0/2327107565 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1227313725 0 0) 0x7fbc100aa7b0 con 0x7fbc100ae250 2024-03-29T11:27:49.033 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.038+0000 7fbc0f7fe640 1 -- 172.21.15.17:0/2327107565 >> v1:172.21.15.94:6789/0 conn(0x7fbc1014bba0 legacy=0x7fbc1014df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:49.033 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.038+0000 7fbc0f7fe640 1 -- 172.21.15.17:0/2327107565 >> v1:172.21.15.17:6789/0 conn(0x7fbc100a9720 legacy=0x7fbc1013f7d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:49.033 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.040+0000 7fbc0f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:27:49.033 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.040+0000 7fbc0f7fe640 1 monclient: found mon.noname-b 2024-03-29T11:27:49.033 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.040+0000 7fbc0f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:49.033 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.040+0000 7fbc0f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.17:6790/0 2024-03-29T11:27:49.033 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.040+0000 7fbc0f7fe640 1 -- 172.21.15.17:0/2327107565 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc10143280 con 0x7fbc100ae250 2024-03-29T11:27:49.033 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.040+0000 7fbc0f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:27:49.033 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.040+0000 7fbc0f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:27:49.033 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.040+0000 7fbc0f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:49.033 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.040+0000 7fbc0f7fe640 1 -- 172.21.15.17:0/2327107565 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fbc04001670 con 0x7fbc100ae250 2024-03-29T11:27:49.033 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.040+0000 7fbc0f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:49.033 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.040+0000 7fbc0f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:49.034 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.040+0000 7fbc0f7fe640 10 monclient: dump: 2024-03-29T11:27:49.034 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:27:49.034 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:27:49.034 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:49.034 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:49.034 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:49.034 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:49.034 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:27:49.034 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:27:49.034 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:27:49.034 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:49.034 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.041+0000 7fbc16c4e640 5 monclient: authenticate success, global_id 4589 2024-03-29T11:27:49.035 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.041+0000 7fbc16c4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:49.035 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.041+0000 7fbc0f7fe640 1 -- 172.21.15.17:0/2327107565 <== mon.2 v1:172.21.15.17:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc040017d0 con 0x7fbc100ae250 2024-03-29T11:27:49.035 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.041+0000 7fbc0f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:49.035 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.041+0000 7fbc0f7fe640 1 -- 172.21.15.17:0/2327107565 <== mon.2 v1:172.21.15.17:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fbc04001ba0 con 0x7fbc100ae250 2024-03-29T11:27:49.035 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.041+0000 7fbc0f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:49.035 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.041+0000 7fbc0f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:49.035 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.041+0000 7fbc0f7fe640 10 monclient: dump: 2024-03-29T11:27:49.035 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:27:49.035 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:27:49.035 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:49.035 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:49.035 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:49.035 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:49.035 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:27:49.036 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:27:49.036 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:27:49.036 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:49.036 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.041+0000 7fbc16c4e640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:49.036 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.041+0000 7fbc16c4e640 4 set_mon_vals no callback set 2024-03-29T11:27:49.036 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.042+0000 7fbc16c4e640 10 monclient: shutdown 2024-03-29T11:27:49.036 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.042+0000 7fbc16c4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:49.036 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.042+0000 7fbc16c4e640 1 -- 172.21.15.17:0/2327107565 >> v1:172.21.15.17:6790/0 conn(0x7fbc100ae250 legacy=0x7fbc10142b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:49.036 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.042+0000 7fbc16c4e640 1 -- 172.21.15.17:0/2327107565 shutdown_connections 2024-03-29T11:27:49.036 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.042+0000 7fbc16c4e640 1 -- 172.21.15.17:0/2327107565 >> 172.21.15.17:0/2327107565 conn(0x7fbc100b35e0 msgr2=0x7fbc100b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:49.036 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.042+0000 7fbc16c4e640 1 -- 172.21.15.17:0/2327107565 shutdown_connections 2024-03-29T11:27:49.036 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.042+0000 7fbc16c4e640 1 -- 172.21.15.17:0/2327107565 wait complete. 2024-03-29T11:27:49.036 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.043+0000 7fbc16c4e640 10 monclient: build_initial_monmap 2024-03-29T11:27:49.036 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.043+0000 7fbc16c4e640 10 monclient: monmap: 2024-03-29T11:27:49.036 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:27:49.037 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:49.037 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:27:49.037 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:27:49.037 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:49.037 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:49.037 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:27:49.037 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:27:49.037 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:27:49.037 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:49.045 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.044+0000 7fbc16c4e640 1 librados: starting msgr at 2024-03-29T11:27:49.045 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.044+0000 7fbc16c4e640 1 librados: starting objecter 2024-03-29T11:27:49.045 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.044+0000 7fbc16c4e640 1 Processor -- start 2024-03-29T11:27:49.045 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.044+0000 7fbc16c4e640 1 -- start start 2024-03-29T11:27:49.045 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.044+0000 7fbc16c4e640 1 librados: setting wanted keys 2024-03-29T11:27:49.045 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.044+0000 7fbc16c4e640 1 librados: calling monclient init 2024-03-29T11:27:49.045 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.044+0000 7fbc16c4e640 10 monclient: init 2024-03-29T11:27:49.045 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.045+0000 7fbc16c4e640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:49.045 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.045+0000 7fbc16c4e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:27:49.045 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.045+0000 7fbc16c4e640 10 monclient(hunting): picked mon.noname-c con 0x7fbc100a9720 addr v1:172.21.15.17:6790/0 2024-03-29T11:27:49.045 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.045+0000 7fbc16c4e640 10 monclient(hunting): picked mon.noname-a con 0x7fbc1014bba0 addr v1:172.21.15.17:6789/0 2024-03-29T11:27:49.045 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.046+0000 7fbc16c4e640 10 monclient(hunting): picked mon.noname-b con 0x7fbc1013e8a0 addr v1:172.21.15.94:6789/0 2024-03-29T11:27:49.045 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.046+0000 7fbc16c4e640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc101d73d0 con 0x7fbc1014bba0 2024-03-29T11:27:49.046 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.046+0000 7fbc16c4e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc1013e110 con 0x7fbc1013e8a0 2024-03-29T11:27:49.046 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.046+0000 7fbc16c4e640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc1013ba00 con 0x7fbc100a9720 2024-03-29T11:27:49.046 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.046+0000 7fbc16c4e640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:49.046 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.046+0000 7fbc16c4e640 10 monclient(hunting): authenticate will time out at 1375.250918s 2024-03-29T11:27:49.046 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.046+0000 7fbc149ea640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7fbc100a9720 0x7fbc1013c1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:57216/0 (socket says 172.21.15.17:57216) 2024-03-29T11:27:49.046 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.046+0000 7fbc149ea640 1 -- 172.21.15.17:0/1982371340 learned_addr learned my addr 172.21.15.17:0/1982371340 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:27:49.046 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.046+0000 7fbc0d7fa640 10 client.?.objecter ms_handle_connect 0x7fbc100a9720 2024-03-29T11:27:49.046 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.046+0000 7fbc0d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:49.046 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.046+0000 7fbc0d7fa640 1 -- 172.21.15.17:0/1982371340 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3381857161 0 0) 0x7fbc1013ba00 con 0x7fbc100a9720 2024-03-29T11:27:49.046 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.046+0000 7fbc0d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:27:49.046 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.046+0000 7fbc0d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:49.046 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.046+0000 7fbc0d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:49.046 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.046+0000 7fbc0d7fa640 10 monclient(hunting): my global_id is 4592 2024-03-29T11:27:49.046 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.046+0000 7fbc0d7fa640 1 -- 172.21.15.17:0/1982371340 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbf4004eb0 con 0x7fbc100a9720 2024-03-29T11:27:49.047 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.047+0000 7fbc0d7fa640 10 client.?.objecter ms_handle_connect 0x7fbc1013e8a0 2024-03-29T11:27:49.047 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.047+0000 7fbc0d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:49.047 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.047+0000 7fbc0d7fa640 1 -- 172.21.15.17:0/1982371340 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2765086823 0 0) 0x7fbbf4004eb0 con 0x7fbc100a9720 2024-03-29T11:27:49.047 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.047+0000 7fbc0d7fa640 1 -- 172.21.15.17:0/1982371340 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc1013ba00 con 0x7fbc100a9720 2024-03-29T11:27:49.047 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.047+0000 7fbc0d7fa640 1 -- 172.21.15.17:0/1982371340 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fbc04001ae0 con 0x7fbc100a9720 2024-03-29T11:27:49.047 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.047+0000 7fbc0d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:49.047 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.047+0000 7fbc0d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:49.047 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.047+0000 7fbc0d7fa640 10 monclient(hunting): dump: 2024-03-29T11:27:49.047 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:27:49.047 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:27:49.047 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:49.047 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:49.047 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:49.047 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:49.047 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:27:49.048 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:27:49.048 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:27:49.048 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:49.048 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.047+0000 7fbc0d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:49.048 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.047+0000 7fbc0d7fa640 1 -- 172.21.15.17:0/1982371340 <== mon.2 v1:172.21.15.17:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1446172982 0 0) 0x7fbc1013ba00 con 0x7fbc100a9720 2024-03-29T11:27:49.048 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.047+0000 7fbc0d7fa640 1 -- 172.21.15.17:0/1982371340 >> v1:172.21.15.94:6789/0 conn(0x7fbc1013e8a0 legacy=0x7fbc101e1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:49.048 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.047+0000 7fbc0d7fa640 1 -- 172.21.15.17:0/1982371340 >> v1:172.21.15.17:6789/0 conn(0x7fbc1014bba0 legacy=0x7fbc101d7860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:27:49.048 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.050+0000 7fbc0d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:27:49.048 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.051+0000 7fbc0d7fa640 1 monclient: found mon.c 2024-03-29T11:27:49.048 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.051+0000 7fbc0d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:49.048 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.051+0000 7fbc0d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:27:49.048 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.051+0000 7fbc0d7fa640 1 -- 172.21.15.17:0/1982371340 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc101d7f70 con 0x7fbc100a9720 2024-03-29T11:27:49.048 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.051+0000 7fbc0d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:27:49.048 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.051+0000 7fbc0d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:27:49.048 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.051+0000 7fbc0d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:49.049 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.051+0000 7fbc16c4e640 5 monclient: authenticate success, global_id 4592 2024-03-29T11:27:49.049 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.051+0000 7fbc16c4e640 10 monclient: _renew_subs 2024-03-29T11:27:49.049 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.051+0000 7fbc16c4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:27:49.049 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.051+0000 7fbc16c4e640 1 -- 172.21.15.17:0/1982371340 --> v1:172.21.15.17:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc1013cf20 con 0x7fbc100a9720 2024-03-29T11:27:49.049 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.051+0000 7fbc16c4e640 10 client.4592.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:49.049 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.051+0000 7fbc16c4e640 10 monclient: _renew_subs 2024-03-29T11:27:49.049 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.051+0000 7fbc16c4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:27:49.049 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.051+0000 7fbc0d7fa640 1 -- 172.21.15.17:0/1982371340 <== mon.2 v1:172.21.15.17:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc04004b00 con 0x7fbc100a9720 2024-03-29T11:27:49.049 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.051+0000 7fbc16c4e640 1 -- 172.21.15.17:0/1982371340 --> v1:172.21.15.17:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc101e3470 con 0x7fbc100a9720 2024-03-29T11:27:49.049 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.051+0000 7fbc16c4e640 1 librados: init done 2024-03-29T11:27:49.049 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.051+0000 7fbc0d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:49.049 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.051+0000 7fbc0d7fa640 1 -- 172.21.15.17:0/1982371340 <== mon.2 v1:172.21.15.17:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fbc04004f10 con 0x7fbc100a9720 2024-03-29T11:27:49.049 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.051+0000 7fbc0d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:49.049 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.051+0000 7fbc0d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:49.049 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.051+0000 7fbc0d7fa640 10 monclient: dump: 2024-03-29T11:27:49.050 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:27:49.050 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:27:49.050 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:49.050 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:49.050 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:49.050 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:49.050 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:27:49.050 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:27:49.050 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:27:49.050 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:49.050 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.051+0000 7fbc0effd640 4 set_mon_vals no callback set 2024-03-29T11:27:49.050 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.052+0000 7fbc16c4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:49.050 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.052+0000 7fbc16c4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:49.050 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.052+0000 7fbc16c4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:27:49.051 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.052+0000 7fbc16c4e640 1 -- 172.21.15.17:0/1982371340 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc100b2990 con 0x7fbc100a9720 2024-03-29T11:27:49.051 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.053+0000 7fbc0d7fa640 1 -- 172.21.15.17:0/1982371340 <== mon.2 v1:172.21.15.17:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7fbc04005070 con 0x7fbc100a9720 2024-03-29T11:27:49.051 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.054+0000 7fbc149ea640 1 -- 172.21.15.17:0/1982371340 >> v1:172.21.15.17:6790/0 conn(0x7fbc100a9720 legacy=0x7fbc1013c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-29T11:27:49.051 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.054+0000 7fbc149ea640 1 -- 172.21.15.17:0/1982371340 >> v1:172.21.15.17:6790/0 conn(0x7fbc100a9720 legacy=0x7fbc1013c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:27:49.051 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.054+0000 7fbc149ea640 1 --1- 172.21.15.17:0/1982371340 >> v1:172.21.15.17:6790/0 conn(0x7fbc100a9720 0x7fbc1013c1e0 :-1 s=OPENED pgs=877397 cs=1 l=1).handle_message read tag failed 2024-03-29T11:27:49.051 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.054+0000 7fbc149ea640 1 --1- 172.21.15.17:0/1982371340 >> v1:172.21.15.17:6790/0 conn(0x7fbc100a9720 0x7fbc1013c1e0 :-1 s=OPENED pgs=877397 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:27:49.051 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.054+0000 7fbc0d7fa640 10 monclient: ms_handle_reset current mon v1:172.21.15.17:6790/0 2024-03-29T11:27:49.051 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.054+0000 7fbc0d7fa640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:49.051 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.054+0000 7fbc0d7fa640 1 -- 172.21.15.17:0/1982371340 >> v1:172.21.15.17:6790/0 conn(0x7fbc100a9720 legacy=0x7fbc1013c1e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:27:49.051 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.054+0000 7fbc0d7fa640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:27:49.051 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.054+0000 7fbc0d7fa640 10 monclient(hunting): picked mon.a con 0x7fbc1014bba0 addr v1:172.21.15.17:6789/0 2024-03-29T11:27:49.051 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.054+0000 7fbc0d7fa640 10 monclient(hunting): picked mon.b con 0x7fbc1013e8a0 addr v1:172.21.15.94:6789/0 2024-03-29T11:27:49.051 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.054+0000 7fbc0d7fa640 10 monclient(hunting): picked mon.c con 0x7fbbf4078ec0 addr v1:172.21.15.17:6790/0 2024-03-29T11:27:49.051 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.054+0000 7fbc0d7fa640 1 -- 172.21.15.17:0/1982371340 --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbc1013ba00 con 0x7fbc1014bba0 2024-03-29T11:27:49.051 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.054+0000 7fbc0d7fa640 1 -- 172.21.15.17:0/1982371340 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbc101d73d0 con 0x7fbc1013e8a0 2024-03-29T11:27:49.052 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.054+0000 7fbc0d7fa640 1 -- 172.21.15.17:0/1982371340 --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbbf4004eb0 con 0x7fbbf4078ec0 2024-03-29T11:27:49.052 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.054+0000 7fbc0d7fa640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:49.052 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.054+0000 7fbc0d7fa640 10 client.4592.objecter ms_handle_connect 0x7fbc1013e8a0 2024-03-29T11:27:49.052 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.054+0000 7fbc0d7fa640 10 client.4592.objecter resend_mon_ops 2024-03-29T11:27:49.052 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.054+0000 7fbc149ea640 10 monclient(hunting): get_auth_request con 0x7fbbf406dc50 auth_method 0 2024-03-29T11:27:49.052 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.054+0000 7fbc0d7fa640 10 client.4592.objecter ms_handle_connect 0x7fbbf4078ec0 2024-03-29T11:27:49.052 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.054+0000 7fbc0d7fa640 10 client.4592.objecter resend_mon_ops 2024-03-29T11:27:49.052 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.055+0000 7fbc0d7fa640 10 client.4592.objecter ms_handle_connect 0x7fbbf406dc50 2024-03-29T11:27:49.056 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.057+0000 7fbc0d7fa640 10 client.4592.objecter ms_handle_connect 0x7fbc1014bba0 2024-03-29T11:27:49.056 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.057+0000 7fbc0d7fa640 10 client.4592.objecter resend_mon_ops 2024-03-29T11:27:49.057 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.057+0000 7fbc0d7fa640 1 -- 172.21.15.17:0/1982371340 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 560091445 0 0) 0x7fbc101d73d0 con 0x7fbc1013e8a0 2024-03-29T11:27:49.057 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.057+0000 7fbc0d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:27:49.057 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.057+0000 7fbc0d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:49.057 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.057+0000 7fbc0d7fa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T11:27:49.057 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.057+0000 7fbc0d7fa640 1 -- 172.21.15.17:0/1982371340 --> v1:172.21.15.94:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fbbf407f010 con 0x7fbc1013e8a0 2024-03-29T11:27:49.057 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.057+0000 7fbc0d7fa640 1 -- 172.21.15.17:0/1982371340 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1822956639 0 0) 0x7fbc1013ba00 con 0x7fbc1014bba0 2024-03-29T11:27:49.057 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.057+0000 7fbc0d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:27:49.057 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.057+0000 7fbc0d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:49.057 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.057+0000 7fbc0d7fa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T11:27:49.057 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.057+0000 7fbc0d7fa640 1 -- 172.21.15.17:0/1982371340 --> v1:172.21.15.17:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fbc101d73d0 con 0x7fbc1014bba0 2024-03-29T11:27:49.057 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.057+0000 7fbc0d7fa640 1 -- 172.21.15.17:0/1982371340 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 3639499544 0 0) 0x7fbbf407f010 con 0x7fbc1013e8a0 2024-03-29T11:27:49.057 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.057+0000 7fbc0d7fa640 1 -- 172.21.15.17:0/1982371340 >> v1:172.21.15.17:6790/0 conn(0x7fbbf4078ec0 legacy=0x7fbbf407b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:49.057 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.057+0000 7fbc0d7fa640 1 -- 172.21.15.17:0/1982371340 >> v1:172.21.15.17:6789/0 conn(0x7fbc1014bba0 legacy=0x7fbbf4073910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:49.057 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.057+0000 7fbc0d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:27:49.058 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.057+0000 7fbc0d7fa640 1 monclient: found mon.b 2024-03-29T11:27:49.058 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.057+0000 7fbc0d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:49.058 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.057+0000 7fbc0d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:27:49.058 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.057+0000 7fbc0d7fa640 1 -- 172.21.15.17:0/1982371340 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7fbbf4080040 con 0x7fbc1013e8a0 2024-03-29T11:27:49.058 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.057+0000 7fbc0d7fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:49.058 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.057+0000 7fbc0d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:27:49.058 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.057+0000 7fbc0d7fa640 1 -- 172.21.15.17:0/1982371340 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbf407f570 con 0x7fbc1013e8a0 2024-03-29T11:27:49.059 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.058+0000 7fbc0d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:27:49.059 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.058+0000 7fbc0d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:27:49.059 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.058+0000 7fbc0d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:49.059 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.058+0000 7fbc0d7fa640 1 -- 172.21.15.17:0/1982371340 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fbc08001980 con 0x7fbc1013e8a0 2024-03-29T11:27:49.059 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.058+0000 7fbc0d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:49.059 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.058+0000 7fbc0d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:49.059 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.058+0000 7fbc0d7fa640 10 monclient: dump: 2024-03-29T11:27:49.059 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:27:49.059 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:27:49.059 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:49.059 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:49.059 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:49.059 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:49.060 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:27:49.060 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:27:49.060 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:27:49.060 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:49.060 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.058+0000 7fbc0d7fa640 1 -- 172.21.15.17:0/1982371340 <== mon.1 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc08001720 con 0x7fbc1013e8a0 2024-03-29T11:27:49.060 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.058+0000 7fbc0d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:49.060 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.058+0000 7fbc0e7fc640 4 set_mon_vals no callback set 2024-03-29T11:27:49.060 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.060+0000 7fbc0d7fa640 1 -- 172.21.15.17:0/1982371340 <== mon.1 v1:172.21.15.94:6789/0 5 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7fbc08001ae0 con 0x7fbc1013e8a0 2024-03-29T11:27:49.060 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.061+0000 7fbc0d7fa640 1 -- 172.21.15.17:0/1982371340 <== mon.1 v1:172.21.15.94:6789/0 6 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7fbc08082f70 con 0x7fbc1013e8a0 2024-03-29T11:27:49.060 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.061+0000 7fbc0d7fa640 10 client.4592.objecter ms_dispatch 0x7fbc10000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:49.061 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.061+0000 7fbc0d7fa640 3 client.4592.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:49.061 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.061+0000 7fbc0d7fa640 3 client.4592.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:49.061 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.061+0000 7fbc0d7fa640 20 client.4592.objecter dump_active .. 0 homeless 2024-03-29T11:27:49.061 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.066+0000 7fbc0d7fa640 1 -- 172.21.15.17:0/1982371340 <== mon.1 v1:172.21.15.94:6789/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbc08051fa0 con 0x7fbc1013e8a0 2024-03-29T11:27:49.061 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.066+0000 7fbc0d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:49.061 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.066+0000 7fbc0d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:49.375 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.386+0000 7fbc16c4e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:49.375 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.386+0000 7fbc16c4e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:49.375 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.386+0000 7fbc16c4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:27:49.375 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.386+0000 7fbc16c4e640 1 -- 172.21.15.17:0/1982371340 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbc10075f10 con 0x7fbc1013e8a0 2024-03-29T11:27:49.376 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.387+0000 7fbc0d7fa640 1 -- 172.21.15.17:0/1982371340 <== mon.1 v1:172.21.15.94:6789/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7fbc08051fa0 con 0x7fbc1013e8a0 2024-03-29T11:27:49.376 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.387+0000 7fbc0d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:49.376 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.387+0000 7fbc0d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:49.376 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:27:49.376 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:49.381 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.390+0000 7fbc16c4e640 10 librados: watch_flush enter 2024-03-29T11:27:49.382 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.390+0000 7fbc16c4e640 10 librados: watch_flush exit 2024-03-29T11:27:49.382 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.390+0000 7fbc16c4e640 20 client.4592.objecter shutdown clearing up homeless session... 2024-03-29T11:27:49.382 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.390+0000 7fbc16c4e640 10 client.4592.objecter successfully canceled tick 2024-03-29T11:27:49.382 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.390+0000 7fbc16c4e640 1 -- 172.21.15.17:0/1982371340 >> v1:172.21.15.17:6816/3236185052 conn(0x7fbbf406dc50 legacy=0x7fbbf40700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:49.382 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.390+0000 7fbc16c4e640 10 monclient: shutdown 2024-03-29T11:27:49.382 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.390+0000 7fbc16c4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:49.382 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.390+0000 7fbc16c4e640 1 -- 172.21.15.17:0/1982371340 >> v1:172.21.15.94:6789/0 conn(0x7fbc1013e8a0 legacy=0x7fbbf4077360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:49.383 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.390+0000 7fbc16c4e640 1 -- 172.21.15.17:0/1982371340 shutdown_connections 2024-03-29T11:27:49.383 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.390+0000 7fbc16c4e640 1 -- 172.21.15.17:0/1982371340 >> 172.21.15.17:0/1982371340 conn(0x7fbc100b35e0 msgr2=0x7fbc100b2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:49.383 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.390+0000 7fbc16c4e640 1 -- 172.21.15.17:0/1982371340 shutdown_connections 2024-03-29T11:27:49.383 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.390+0000 7fbc16c4e640 1 -- 172.21.15.17:0/1982371340 wait complete. 2024-03-29T11:27:49.383 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:49.391+0000 7fbc16c4e640 1 librados: shutdown 2024-03-29T11:27:49.392 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:56.199 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:56.336 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.347+0000 7f046bb48640 10 monclient: get_monmap_and_config 2024-03-29T11:27:56.336 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.347+0000 7f046bb48640 10 monclient: build_initial_monmap 2024-03-29T11:27:56.336 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.347+0000 7f046bb48640 10 monclient: monmap: 2024-03-29T11:27:56.336 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:27:56.336 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:27:56.336 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:27:56.348429+0000 2024-03-29T11:27:56.336 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:27:56.348429+0000 2024-03-29T11:27:56.337 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:56.337 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:56.337 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:27:56.337 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:27:56.337 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:27:56.337 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:56.340 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.348+0000 7f046bb48640 1 Processor -- start 2024-03-29T11:27:56.340 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.348+0000 7f046bb48640 1 -- start start 2024-03-29T11:27:56.340 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.348+0000 7f046bb48640 10 monclient: init 2024-03-29T11:27:56.340 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.349+0000 7f046bb48640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:56.340 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.349+0000 7f046bb48640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:27:56.341 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.349+0000 7f046bb48640 10 monclient(hunting): picked mon.noname-c con 0x7f04641457f0 addr v1:172.21.15.94:6789/0 2024-03-29T11:27:56.341 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.349+0000 7f046bb48640 10 monclient(hunting): picked mon.noname-b con 0x7f046413f650 addr v1:172.21.15.17:6790/0 2024-03-29T11:27:56.341 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.349+0000 7f046bb48640 10 monclient(hunting): picked mon.noname-a con 0x7f04640a9d70 addr v1:172.21.15.17:6789/0 2024-03-29T11:27:56.341 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.349+0000 7f046bb48640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04640aaef0 con 0x7f04640a9d70 2024-03-29T11:27:56.341 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.349+0000 7f046bb48640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0464145bd0 con 0x7f04641457f0 2024-03-29T11:27:56.341 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.349+0000 7f046bb48640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0464145eb0 con 0x7f046413f650 2024-03-29T11:27:56.341 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.349+0000 7f046bb48640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:56.341 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.349+0000 7f046bb48640 10 monclient(hunting): authenticate will time out at 1382.554415s 2024-03-29T11:27:56.341 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.349+0000 7f04698e4640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f04641457f0 0x7f046413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:35424/0 (socket says 172.21.15.17:35424) 2024-03-29T11:27:56.341 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.349+0000 7f04698e4640 1 -- 172.21.15.17:0/3757918679 learned_addr learned my addr 172.21.15.17:0/3757918679 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:27:56.342 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.350+0000 7f04688e2640 1 -- 172.21.15.17:0/3757918679 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2831784933 0 0) 0x7f0464145bd0 con 0x7f04641457f0 2024-03-29T11:27:56.342 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.350+0000 7f04688e2640 10 monclient(hunting): _negotiate 2024-03-29T11:27:56.342 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.350+0000 7f04688e2640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:56.342 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.350+0000 7f04688e2640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:56.342 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.350+0000 7f04688e2640 10 monclient(hunting): my global_id is 4561 2024-03-29T11:27:56.342 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.350+0000 7f04688e2640 1 -- 172.21.15.17:0/3757918679 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0450003520 con 0x7f04641457f0 2024-03-29T11:27:56.342 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.350+0000 7f04688e2640 1 -- 172.21.15.17:0/3757918679 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1395719038 0 0) 0x7f0464145eb0 con 0x7f046413f650 2024-03-29T11:27:56.342 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.350+0000 7f04688e2640 10 monclient(hunting): _negotiate 2024-03-29T11:27:56.342 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.350+0000 7f04688e2640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:56.342 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.350+0000 7f04688e2640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:56.342 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.351+0000 7f04688e2640 10 monclient(hunting): my global_id is 4595 2024-03-29T11:27:56.342 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.351+0000 7f04688e2640 1 -- 172.21.15.17:0/3757918679 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0464145bd0 con 0x7f046413f650 2024-03-29T11:27:56.343 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.351+0000 7f04688e2640 1 -- 172.21.15.17:0/3757918679 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3654796153 0 0) 0x7f0464145bd0 con 0x7f046413f650 2024-03-29T11:27:56.343 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.351+0000 7f04688e2640 1 -- 172.21.15.17:0/3757918679 >> v1:172.21.15.94:6789/0 conn(0x7f04641457f0 legacy=0x7f046413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:56.343 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.351+0000 7f04688e2640 1 -- 172.21.15.17:0/3757918679 >> v1:172.21.15.17:6789/0 conn(0x7f04640a9d70 legacy=0x7f046414d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:56.347 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.352+0000 7f04688e2640 10 monclient: _finish_hunting 0 2024-03-29T11:27:56.347 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.352+0000 7f04688e2640 1 monclient: found mon.noname-b 2024-03-29T11:27:56.347 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.352+0000 7f04688e2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:56.347 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.352+0000 7f04688e2640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.17:6790/0 2024-03-29T11:27:56.347 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.352+0000 7f04688e2640 1 -- 172.21.15.17:0/3757918679 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0464146340 con 0x7f046413f650 2024-03-29T11:27:56.347 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.353+0000 7f04688e2640 10 monclient: _finish_auth 0 2024-03-29T11:27:56.348 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.353+0000 7f04688e2640 10 monclient: _check_auth_tickets 2024-03-29T11:27:56.348 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.353+0000 7f04688e2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:56.348 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.353+0000 7f04688e2640 1 -- 172.21.15.17:0/3757918679 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f0454001670 con 0x7f046413f650 2024-03-29T11:27:56.348 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.353+0000 7f046bb48640 5 monclient: authenticate success, global_id 4595 2024-03-29T11:27:56.348 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.353+0000 7f046bb48640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:56.348 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.353+0000 7f04688e2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:56.348 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.353+0000 7f04688e2640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:56.348 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.353+0000 7f04688e2640 10 monclient: dump: 2024-03-29T11:27:56.348 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:27:56.348 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:27:56.348 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:56.348 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:56.348 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:56.348 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:56.348 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:27:56.349 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:27:56.349 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:27:56.349 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:56.349 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.353+0000 7f046bb48640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:56.349 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.353+0000 7f04688e2640 1 -- 172.21.15.17:0/3757918679 <== mon.2 v1:172.21.15.17:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0454001890 con 0x7f046413f650 2024-03-29T11:27:56.349 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.353+0000 7f04688e2640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:56.349 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.353+0000 7f04688e2640 1 -- 172.21.15.17:0/3757918679 <== mon.2 v1:172.21.15.17:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f0454001c20 con 0x7f046413f650 2024-03-29T11:27:56.349 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.353+0000 7f04688e2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:56.349 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.353+0000 7f04688e2640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:56.349 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.353+0000 7f04688e2640 10 monclient: dump: 2024-03-29T11:27:56.349 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:27:56.349 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:27:56.349 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:56.349 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:56.349 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:56.350 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:56.350 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:27:56.350 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:27:56.350 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:27:56.350 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:56.350 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.354+0000 7f046bb48640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:56.350 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.354+0000 7f046bb48640 4 set_mon_vals no callback set 2024-03-29T11:27:56.350 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.355+0000 7f046bb48640 10 monclient: shutdown 2024-03-29T11:27:56.350 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.355+0000 7f046bb48640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:56.350 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.355+0000 7f046bb48640 1 -- 172.21.15.17:0/3757918679 >> v1:172.21.15.17:6790/0 conn(0x7f046413f650 legacy=0x7f04640b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:56.350 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.355+0000 7f046bb48640 1 -- 172.21.15.17:0/3757918679 shutdown_connections 2024-03-29T11:27:56.351 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.355+0000 7f046bb48640 1 -- 172.21.15.17:0/3757918679 >> 172.21.15.17:0/3757918679 conn(0x7f0464066a00 msgr2=0x7f0464066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:56.351 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.355+0000 7f046bb48640 1 -- 172.21.15.17:0/3757918679 shutdown_connections 2024-03-29T11:27:56.351 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.355+0000 7f046bb48640 1 -- 172.21.15.17:0/3757918679 wait complete. 2024-03-29T11:27:56.351 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.356+0000 7f046bb48640 10 monclient: build_initial_monmap 2024-03-29T11:27:56.351 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.356+0000 7f046bb48640 10 monclient: monmap: 2024-03-29T11:27:56.351 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:27:56.351 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:56.351 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:27:56.351 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:27:56.351 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:56.351 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:56.351 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:27:56.351 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:27:56.351 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:27:56.351 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:56.352 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.357+0000 7f046bb48640 1 librados: starting msgr at 2024-03-29T11:27:56.352 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.357+0000 7f046bb48640 1 librados: starting objecter 2024-03-29T11:27:56.352 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.357+0000 7f046bb48640 1 Processor -- start 2024-03-29T11:27:56.352 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.357+0000 7f046bb48640 1 -- start start 2024-03-29T11:27:56.352 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.357+0000 7f046bb48640 1 librados: setting wanted keys 2024-03-29T11:27:56.352 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.357+0000 7f046bb48640 1 librados: calling monclient init 2024-03-29T11:27:56.352 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.357+0000 7f046bb48640 10 monclient: init 2024-03-29T11:27:56.352 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.357+0000 7f046bb48640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:56.352 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.357+0000 7f046bb48640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:27:56.352 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.357+0000 7f046bb48640 10 monclient(hunting): picked mon.noname-b con 0x7f04640a9d70 addr v1:172.21.15.94:6789/0 2024-03-29T11:27:56.352 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.357+0000 7f046bb48640 10 monclient(hunting): picked mon.noname-c con 0x7f046413f650 addr v1:172.21.15.17:6790/0 2024-03-29T11:27:56.352 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.357+0000 7f046bb48640 10 monclient(hunting): picked mon.noname-a con 0x7f04641457f0 addr v1:172.21.15.17:6789/0 2024-03-29T11:27:56.352 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.358+0000 7f046bb48640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0464142d60 con 0x7f04641457f0 2024-03-29T11:27:56.352 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.358+0000 7f046bb48640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04641d7070 con 0x7f04640a9d70 2024-03-29T11:27:56.352 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.358+0000 7f046bb48640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04641d7ea0 con 0x7f046413f650 2024-03-29T11:27:56.352 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.358+0000 7f046bb48640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:56.353 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.358+0000 7f046bb48640 10 monclient(hunting): authenticate will time out at 1382.562911s 2024-03-29T11:27:56.353 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.358+0000 7f04690e3640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f046413f650 0x7f0464140110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:36676/0 (socket says 172.21.15.17:36676) 2024-03-29T11:27:56.353 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.358+0000 7f04690e3640 1 -- 172.21.15.17:0/1968963552 learned_addr learned my addr 172.21.15.17:0/1968963552 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:27:56.353 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.358+0000 7f045a7fc640 10 client.?.objecter ms_handle_connect 0x7f046413f650 2024-03-29T11:27:56.353 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.358+0000 7f045a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:56.353 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.358+0000 7f045a7fc640 1 -- 172.21.15.17:0/1968963552 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1175388288 0 0) 0x7f04641d7ea0 con 0x7f046413f650 2024-03-29T11:27:56.353 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.358+0000 7f045a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:27:56.353 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.358+0000 7f045a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:56.353 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.358+0000 7f045a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:56.354 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.358+0000 7f045a7fc640 10 monclient(hunting): my global_id is 4598 2024-03-29T11:27:56.354 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.358+0000 7f045a7fc640 1 -- 172.21.15.17:0/1968963552 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0448004eb0 con 0x7f046413f650 2024-03-29T11:27:56.357 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.359+0000 7f045a7fc640 10 client.?.objecter ms_handle_connect 0x7f04640a9d70 2024-03-29T11:27:56.357 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.359+0000 7f045a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:56.357 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.359+0000 7f045a7fc640 1 -- 172.21.15.17:0/1968963552 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3848100152 0 0) 0x7f0448004eb0 con 0x7f046413f650 2024-03-29T11:27:56.357 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.359+0000 7f045a7fc640 1 -- 172.21.15.17:0/1968963552 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04641d7ea0 con 0x7f046413f650 2024-03-29T11:27:56.357 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.359+0000 7f045a7fc640 1 -- 172.21.15.17:0/1968963552 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f0454001ce0 con 0x7f046413f650 2024-03-29T11:27:56.357 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.359+0000 7f045a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:56.358 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.359+0000 7f045a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:56.358 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.359+0000 7f045a7fc640 10 monclient(hunting): dump: 2024-03-29T11:27:56.358 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:27:56.358 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:27:56.358 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:56.358 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:56.358 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:56.358 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:56.358 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:27:56.358 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:27:56.358 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:27:56.358 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:56.358 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.359+0000 7f045a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:56.359 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.359+0000 7f045a7fc640 1 -- 172.21.15.17:0/1968963552 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 414899931 0 0) 0x7f04641d7070 con 0x7f04640a9d70 2024-03-29T11:27:56.359 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.359+0000 7f045a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:27:56.359 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.359+0000 7f045a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:56.359 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.359+0000 7f045a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:56.359 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.359+0000 7f045a7fc640 10 monclient(hunting): my global_id is 4564 2024-03-29T11:27:56.359 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.359+0000 7f045a7fc640 1 -- 172.21.15.17:0/1968963552 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0448004eb0 con 0x7f04640a9d70 2024-03-29T11:27:56.359 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.359+0000 7f045a7fc640 1 -- 172.21.15.17:0/1968963552 <== mon.2 v1:172.21.15.17:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2054737239 0 0) 0x7f04641d7ea0 con 0x7f046413f650 2024-03-29T11:27:56.359 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.359+0000 7f045a7fc640 1 -- 172.21.15.17:0/1968963552 >> v1:172.21.15.94:6789/0 conn(0x7f04640a9d70 legacy=0x7f04641d7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:56.359 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.359+0000 7f045a7fc640 1 -- 172.21.15.17:0/1968963552 >> v1:172.21.15.17:6789/0 conn(0x7f04641457f0 legacy=0x7f04641e2990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:27:56.359 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.360+0000 7f045a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:27:56.359 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.360+0000 7f045a7fc640 1 monclient: found mon.c 2024-03-29T11:27:56.359 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.360+0000 7f045a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:56.359 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.360+0000 7f045a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:27:56.359 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.360+0000 7f045a7fc640 1 -- 172.21.15.17:0/1968963552 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04641d8030 con 0x7f046413f650 2024-03-29T11:27:56.359 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.360+0000 7f045a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:27:56.360 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.360+0000 7f045a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:27:56.360 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.360+0000 7f045a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:56.360 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.360+0000 7f046bb48640 5 monclient: authenticate success, global_id 4598 2024-03-29T11:27:56.360 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.360+0000 7f046bb48640 10 monclient: _renew_subs 2024-03-29T11:27:56.360 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.360+0000 7f046bb48640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:27:56.360 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.360+0000 7f046bb48640 1 -- 172.21.15.17:0/1968963552 --> v1:172.21.15.17:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0464144c30 con 0x7f046413f650 2024-03-29T11:27:56.360 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.360+0000 7f046bb48640 10 client.4598.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:56.360 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.360+0000 7f045a7fc640 1 -- 172.21.15.17:0/1968963552 <== mon.2 v1:172.21.15.17:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0454003910 con 0x7f046413f650 2024-03-29T11:27:56.360 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.360+0000 7f046bb48640 10 monclient: _renew_subs 2024-03-29T11:27:56.360 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.360+0000 7f046bb48640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:27:56.360 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.360+0000 7f046bb48640 1 -- 172.21.15.17:0/1968963552 --> v1:172.21.15.17:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f046413df90 con 0x7f046413f650 2024-03-29T11:27:56.360 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.360+0000 7f046bb48640 1 librados: init done 2024-03-29T11:27:56.360 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.360+0000 7f045a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:56.360 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.360+0000 7f045a7fc640 1 -- 172.21.15.17:0/1968963552 <== mon.2 v1:172.21.15.17:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f0454003d20 con 0x7f046413f650 2024-03-29T11:27:56.361 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.360+0000 7f045a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:56.361 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.360+0000 7f045a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:56.361 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.360+0000 7f045a7fc640 10 monclient: dump: 2024-03-29T11:27:56.361 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:27:56.361 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:27:56.361 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:56.361 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:27:56.361 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:56.361 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:27:56.361 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:27:56.361 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:27:56.361 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:27:56.361 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:27:56.361 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.360+0000 7f045bfff640 4 set_mon_vals no callback set 2024-03-29T11:27:56.361 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.362+0000 7f046bb48640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:56.362 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.362+0000 7f046bb48640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:56.362 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.362+0000 7f046bb48640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:27:56.362 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.362+0000 7f045a7fc640 1 -- 172.21.15.17:0/1968963552 <== mon.2 v1:172.21.15.17:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f0454003e80 con 0x7f046413f650 2024-03-29T11:27:56.362 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.362+0000 7f046bb48640 1 -- 172.21.15.17:0/1968963552 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f044c005080 con 0x7f046413f650 2024-03-29T11:27:56.362 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.363+0000 7f04698e4640 10 monclient: get_auth_request con 0x7f044806dc40 auth_method 0 2024-03-29T11:27:56.362 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.363+0000 7f045a7fc640 1 -- 172.21.15.17:0/1968963552 <== mon.2 v1:172.21.15.17:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f0454054ad0 con 0x7f046413f650 2024-03-29T11:27:56.362 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.363+0000 7f045a7fc640 10 client.4598.objecter ms_dispatch 0x7f0464000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:56.362 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.363+0000 7f045a7fc640 3 client.4598.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:56.362 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.363+0000 7f045a7fc640 3 client.4598.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:56.362 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.363+0000 7f045a7fc640 20 client.4598.objecter dump_active .. 0 homeless 2024-03-29T11:27:56.362 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.363+0000 7f045a7fc640 10 client.4598.objecter ms_handle_connect 0x7f044806dc40 2024-03-29T11:27:56.363 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.373+0000 7f045a7fc640 1 -- 172.21.15.17:0/1968963552 <== mon.2 v1:172.21.15.17:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f04540505e0 con 0x7f046413f650 2024-03-29T11:27:56.364 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.373+0000 7f045a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:56.364 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.373+0000 7f045a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:56.683 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:27:56.683 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:56.683 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.693+0000 7f046bb48640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:56.683 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.693+0000 7f046bb48640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:56.683 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.693+0000 7f046bb48640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:27:56.684 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.693+0000 7f046bb48640 1 -- 172.21.15.17:0/1968963552 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f044c005330 con 0x7f046413f650 2024-03-29T11:27:56.684 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.694+0000 7f045a7fc640 1 -- 172.21.15.17:0/1968963552 <== mon.2 v1:172.21.15.17:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f0454052510 con 0x7f046413f650 2024-03-29T11:27:56.684 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.694+0000 7f045a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:56.684 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.694+0000 7f045a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:56.689 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.697+0000 7f046bb48640 10 librados: watch_flush enter 2024-03-29T11:27:56.689 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.697+0000 7f046bb48640 10 librados: watch_flush exit 2024-03-29T11:27:56.689 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.697+0000 7f046bb48640 20 client.4598.objecter shutdown clearing up homeless session... 2024-03-29T11:27:56.690 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.697+0000 7f046bb48640 10 client.4598.objecter successfully canceled tick 2024-03-29T11:27:56.690 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.697+0000 7f046bb48640 1 -- 172.21.15.17:0/1968963552 >> v1:172.21.15.17:6816/3236185052 conn(0x7f044806dc40 legacy=0x7f0448070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:56.690 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.697+0000 7f046bb48640 10 monclient: shutdown 2024-03-29T11:27:56.690 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.697+0000 7f046bb48640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:56.690 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.697+0000 7f046bb48640 1 -- 172.21.15.17:0/1968963552 >> v1:172.21.15.17:6790/0 conn(0x7f046413f650 legacy=0x7f0464140110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:56.690 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.697+0000 7f046bb48640 1 -- 172.21.15.17:0/1968963552 shutdown_connections 2024-03-29T11:27:56.690 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.697+0000 7f046bb48640 1 -- 172.21.15.17:0/1968963552 >> 172.21.15.17:0/1968963552 conn(0x7f0464066a00 msgr2=0x7f04640b4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:56.691 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.697+0000 7f046bb48640 1 -- 172.21.15.17:0/1968963552 shutdown_connections 2024-03-29T11:27:56.691 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.698+0000 7f046bb48640 1 -- 172.21.15.17:0/1968963552 wait complete. 2024-03-29T11:27:56.691 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:27:56.698+0000 7f046bb48640 1 librados: shutdown 2024-03-29T11:27:56.703 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:03.511 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:03.662 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.672+0000 7f1c702d3640 10 monclient: get_monmap_and_config 2024-03-29T11:28:03.662 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.672+0000 7f1c702d3640 10 monclient: build_initial_monmap 2024-03-29T11:28:03.662 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.672+0000 7f1c702d3640 10 monclient: monmap: 2024-03-29T11:28:03.662 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:28:03.663 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:28:03.663 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:28:03.673855+0000 2024-03-29T11:28:03.663 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:28:03.673855+0000 2024-03-29T11:28:03.663 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:03.663 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:03.663 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:28:03.663 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:28:03.663 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:28:03.663 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:03.664 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.673+0000 7f1c702d3640 1 Processor -- start 2024-03-29T11:28:03.664 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.674+0000 7f1c702d3640 1 -- start start 2024-03-29T11:28:03.664 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.674+0000 7f1c702d3640 10 monclient: init 2024-03-29T11:28:03.664 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.675+0000 7f1c702d3640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:03.664 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.675+0000 7f1c702d3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:28:03.665 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.675+0000 7f1c702d3640 10 monclient(hunting): picked mon.noname-c con 0x7f1c680b0860 addr v1:172.21.15.94:6789/0 2024-03-29T11:28:03.665 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.675+0000 7f1c702d3640 10 monclient(hunting): picked mon.noname-a con 0x7f1c68072450 addr v1:172.21.15.17:6789/0 2024-03-29T11:28:03.665 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.675+0000 7f1c702d3640 10 monclient(hunting): picked mon.noname-b con 0x7f1c6813ee40 addr v1:172.21.15.17:6790/0 2024-03-29T11:28:03.665 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.675+0000 7f1c702d3640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c680ad840 con 0x7f1c68072450 2024-03-29T11:28:03.665 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.675+0000 7f1c702d3640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c6813c830 con 0x7f1c680b0860 2024-03-29T11:28:03.665 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.675+0000 7f1c702d3640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c6813c9c0 con 0x7f1c6813ee40 2024-03-29T11:28:03.665 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.675+0000 7f1c702d3640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:03.665 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.675+0000 7f1c702d3640 10 monclient(hunting): authenticate will time out at 1389.880198s 2024-03-29T11:28:03.666 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.676+0000 7f1c6e870640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f1c6813ee40 0x7f1c6813f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:46908/0 (socket says 172.21.15.17:46908) 2024-03-29T11:28:03.666 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.676+0000 7f1c6e870640 1 -- 172.21.15.17:0/2962688642 learned_addr learned my addr 172.21.15.17:0/2962688642 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:28:03.666 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.677+0000 7f1c6d06d640 1 -- 172.21.15.17:0/2962688642 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1294826675 0 0) 0x7f1c6813c9c0 con 0x7f1c6813ee40 2024-03-29T11:28:03.666 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.677+0000 7f1c6d06d640 10 monclient(hunting): _negotiate 2024-03-29T11:28:03.666 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.677+0000 7f1c6d06d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:03.666 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.677+0000 7f1c6d06d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:03.667 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.677+0000 7f1c6d06d640 10 monclient(hunting): my global_id is 4601 2024-03-29T11:28:03.667 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.677+0000 7f1c6d06d640 1 -- 172.21.15.17:0/2962688642 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c54003520 con 0x7f1c6813ee40 2024-03-29T11:28:03.668 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.678+0000 7f1c6d06d640 1 -- 172.21.15.17:0/2962688642 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3531674272 0 0) 0x7f1c54003520 con 0x7f1c6813ee40 2024-03-29T11:28:03.668 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.678+0000 7f1c6d06d640 1 -- 172.21.15.17:0/2962688642 >> v1:172.21.15.94:6789/0 conn(0x7f1c680b0860 legacy=0x7f1c680acfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:03.668 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.678+0000 7f1c6d06d640 1 -- 172.21.15.17:0/2962688642 >> v1:172.21.15.17:6789/0 conn(0x7f1c68072450 legacy=0x7f1c6813c120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:03.670 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.681+0000 7f1c6d06d640 10 monclient: _finish_hunting 0 2024-03-29T11:28:03.670 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.681+0000 7f1c6d06d640 1 monclient: found mon.noname-b 2024-03-29T11:28:03.670 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.681+0000 7f1c6d06d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:03.670 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.681+0000 7f1c6d06d640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.17:6790/0 2024-03-29T11:28:03.670 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.681+0000 7f1c6d06d640 1 -- 172.21.15.17:0/2962688642 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c6813f970 con 0x7f1c6813ee40 2024-03-29T11:28:03.670 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.681+0000 7f1c6d06d640 10 monclient: _finish_auth 0 2024-03-29T11:28:03.670 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.681+0000 7f1c6d06d640 10 monclient: _check_auth_tickets 2024-03-29T11:28:03.671 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.681+0000 7f1c6d06d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:03.671 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.681+0000 7f1c6d06d640 1 -- 172.21.15.17:0/2962688642 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f1c64001a10 con 0x7f1c6813ee40 2024-03-29T11:28:03.671 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.681+0000 7f1c6d06d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:03.671 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.681+0000 7f1c6d06d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:03.671 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.681+0000 7f1c6d06d640 10 monclient: dump: 2024-03-29T11:28:03.671 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:28:03.671 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:28:03.671 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:03.671 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:03.671 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:03.671 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:03.671 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:28:03.671 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:28:03.671 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:28:03.671 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:03.672 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.681+0000 7f1c702d3640 5 monclient: authenticate success, global_id 4601 2024-03-29T11:28:03.672 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.681+0000 7f1c702d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:03.672 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.681+0000 7f1c6d06d640 1 -- 172.21.15.17:0/2962688642 <== mon.2 v1:172.21.15.17:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c64001be0 con 0x7f1c6813ee40 2024-03-29T11:28:03.672 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.681+0000 7f1c6d06d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:03.673 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.681+0000 7f1c6d06d640 1 -- 172.21.15.17:0/2962688642 <== mon.2 v1:172.21.15.17:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f1c64002030 con 0x7f1c6813ee40 2024-03-29T11:28:03.673 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.681+0000 7f1c6d06d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:03.673 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.681+0000 7f1c6d06d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:03.673 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.681+0000 7f1c6d06d640 10 monclient: dump: 2024-03-29T11:28:03.673 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:28:03.673 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:28:03.673 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:03.673 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:03.673 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:03.673 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:03.673 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:28:03.673 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:28:03.673 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:28:03.673 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:03.674 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.681+0000 7f1c702d3640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:03.674 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.681+0000 7f1c702d3640 4 set_mon_vals no callback set 2024-03-29T11:28:03.674 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.683+0000 7f1c702d3640 10 monclient: shutdown 2024-03-29T11:28:03.674 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.683+0000 7f1c702d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:03.674 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.683+0000 7f1c702d3640 1 -- 172.21.15.17:0/2962688642 >> v1:172.21.15.17:6790/0 conn(0x7f1c6813ee40 legacy=0x7f1c6813f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:03.674 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.683+0000 7f1c702d3640 1 -- 172.21.15.17:0/2962688642 shutdown_connections 2024-03-29T11:28:03.674 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.683+0000 7f1c702d3640 1 -- 172.21.15.17:0/2962688642 >> 172.21.15.17:0/2962688642 conn(0x7f1c6806b660 msgr2=0x7f1c6806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:03.674 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.685+0000 7f1c702d3640 1 -- 172.21.15.17:0/2962688642 shutdown_connections 2024-03-29T11:28:03.674 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.686+0000 7f1c702d3640 1 -- 172.21.15.17:0/2962688642 wait complete. 2024-03-29T11:28:03.677 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.686+0000 7f1c702d3640 10 monclient: build_initial_monmap 2024-03-29T11:28:03.677 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.686+0000 7f1c702d3640 10 monclient: monmap: 2024-03-29T11:28:03.677 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:28:03.677 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:03.677 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:28:03.677 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:28:03.677 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:03.677 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:03.678 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:28:03.678 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:28:03.678 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:28:03.678 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:03.678 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.687+0000 7f1c702d3640 1 librados: starting msgr at 2024-03-29T11:28:03.678 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.687+0000 7f1c702d3640 1 librados: starting objecter 2024-03-29T11:28:03.678 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.687+0000 7f1c702d3640 1 Processor -- start 2024-03-29T11:28:03.678 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.687+0000 7f1c702d3640 1 -- start start 2024-03-29T11:28:03.678 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.687+0000 7f1c702d3640 1 librados: setting wanted keys 2024-03-29T11:28:03.678 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.687+0000 7f1c702d3640 1 librados: calling monclient init 2024-03-29T11:28:03.678 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.687+0000 7f1c702d3640 10 monclient: init 2024-03-29T11:28:03.678 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.688+0000 7f1c702d3640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:03.684 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.688+0000 7f1c702d3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:28:03.684 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.688+0000 7f1c702d3640 10 monclient(hunting): picked mon.noname-a con 0x7f1c68072450 addr v1:172.21.15.17:6789/0 2024-03-29T11:28:03.684 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.688+0000 7f1c702d3640 10 monclient(hunting): picked mon.noname-c con 0x7f1c680b0860 addr v1:172.21.15.17:6790/0 2024-03-29T11:28:03.684 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.688+0000 7f1c702d3640 10 monclient(hunting): picked mon.noname-b con 0x7f1c6813ee40 addr v1:172.21.15.94:6789/0 2024-03-29T11:28:03.684 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.688+0000 7f1c702d3640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c681df070 con 0x7f1c68072450 2024-03-29T11:28:03.684 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.688+0000 7f1c702d3640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c68152530 con 0x7f1c6813ee40 2024-03-29T11:28:03.684 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.688+0000 7f1c702d3640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c68152810 con 0x7f1c680b0860 2024-03-29T11:28:03.685 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.688+0000 7f1c702d3640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:03.685 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.688+0000 7f1c702d3640 10 monclient(hunting): authenticate will time out at 1389.893256s 2024-03-29T11:28:03.685 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.689+0000 7f1c6d86e640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f1c680b0860 0x7f1c68152fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:48764/0 (socket says 172.21.15.17:48764) 2024-03-29T11:28:03.685 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.689+0000 7f1c6d86e640 1 -- 172.21.15.17:0/3415179562 learned_addr learned my addr 172.21.15.17:0/3415179562 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:28:03.685 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.690+0000 7f1c5effd640 10 client.?.objecter ms_handle_connect 0x7f1c680b0860 2024-03-29T11:28:03.685 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.690+0000 7f1c5effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:03.685 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.690+0000 7f1c5effd640 1 -- 172.21.15.17:0/3415179562 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 211323729 0 0) 0x7f1c68152810 con 0x7f1c680b0860 2024-03-29T11:28:03.685 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.690+0000 7f1c5effd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:03.685 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.690+0000 7f1c5effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:03.685 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.690+0000 7f1c5effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:03.685 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.690+0000 7f1c5effd640 10 monclient(hunting): my global_id is 4604 2024-03-29T11:28:03.685 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.690+0000 7f1c5effd640 1 -- 172.21.15.17:0/3415179562 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c4c004eb0 con 0x7f1c680b0860 2024-03-29T11:28:03.685 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.690+0000 7f1c5effd640 1 -- 172.21.15.17:0/3415179562 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3418176578 0 0) 0x7f1c4c004eb0 con 0x7f1c680b0860 2024-03-29T11:28:03.685 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.690+0000 7f1c5effd640 1 -- 172.21.15.17:0/3415179562 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c68152810 con 0x7f1c680b0860 2024-03-29T11:28:03.685 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.690+0000 7f1c5effd640 1 -- 172.21.15.17:0/3415179562 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f1c58001930 con 0x7f1c680b0860 2024-03-29T11:28:03.686 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.690+0000 7f1c5effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:03.686 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.690+0000 7f1c5effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:03.686 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.690+0000 7f1c5effd640 10 monclient(hunting): dump: 2024-03-29T11:28:03.686 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:28:03.686 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:28:03.686 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:03.686 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:03.686 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:03.686 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:03.686 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:28:03.686 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:28:03.686 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:28:03.686 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:03.686 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.690+0000 7f1c5effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:03.687 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.690+0000 7f1c5effd640 1 -- 172.21.15.17:0/3415179562 <== mon.2 v1:172.21.15.17:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 786796921 0 0) 0x7f1c68152810 con 0x7f1c680b0860 2024-03-29T11:28:03.687 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.690+0000 7f1c5effd640 1 -- 172.21.15.17:0/3415179562 >> v1:172.21.15.94:6789/0 conn(0x7f1c6813ee40 legacy=0x7f1c681e6220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:03.687 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.690+0000 7f1c5effd640 1 -- 172.21.15.17:0/3415179562 >> v1:172.21.15.17:6789/0 conn(0x7f1c68072450 legacy=0x7f1c681df500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:03.687 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.691+0000 7f1c5effd640 10 monclient: _finish_hunting 0 2024-03-29T11:28:03.687 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.691+0000 7f1c5effd640 1 monclient: found mon.c 2024-03-29T11:28:03.687 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.691+0000 7f1c5effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:03.687 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.691+0000 7f1c5effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:28:03.687 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.691+0000 7f1c5effd640 1 -- 172.21.15.17:0/3415179562 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c681536b0 con 0x7f1c680b0860 2024-03-29T11:28:03.687 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.691+0000 7f1c5effd640 10 monclient: _finish_auth 0 2024-03-29T11:28:03.687 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.691+0000 7f1c5effd640 10 monclient: _check_auth_tickets 2024-03-29T11:28:03.687 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.691+0000 7f1c5effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:03.687 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.692+0000 7f1c5effd640 1 -- 172.21.15.17:0/3415179562 <== mon.2 v1:172.21.15.17:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c58001c50 con 0x7f1c680b0860 2024-03-29T11:28:03.687 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.692+0000 7f1c5effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:03.687 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.692+0000 7f1c5effd640 1 -- 172.21.15.17:0/3415179562 <== mon.2 v1:172.21.15.17:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f1c58003a40 con 0x7f1c680b0860 2024-03-29T11:28:03.687 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.692+0000 7f1c5effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:03.688 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.692+0000 7f1c5effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:03.688 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.692+0000 7f1c5effd640 10 monclient: dump: 2024-03-29T11:28:03.688 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:28:03.688 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:28:03.688 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:03.688 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:03.688 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:03.688 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:03.688 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:28:03.688 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:28:03.688 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:28:03.688 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:03.688 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.692+0000 7f1c702d3640 5 monclient: authenticate success, global_id 4604 2024-03-29T11:28:03.688 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.692+0000 7f1c702d3640 10 monclient: _renew_subs 2024-03-29T11:28:03.688 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.692+0000 7f1c702d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:28:03.689 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.692+0000 7f1c702d3640 1 -- 172.21.15.17:0/3415179562 --> v1:172.21.15.17:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c681dec70 con 0x7f1c680b0860 2024-03-29T11:28:03.689 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.692+0000 7f1c702d3640 10 client.4604.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:03.689 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.692+0000 7f1c702d3640 10 monclient: _renew_subs 2024-03-29T11:28:03.689 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.692+0000 7f1c702d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:28:03.689 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.692+0000 7f1c702d3640 1 -- 172.21.15.17:0/3415179562 --> v1:172.21.15.17:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c680b29c0 con 0x7f1c680b0860 2024-03-29T11:28:03.689 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.692+0000 7f1c702d3640 1 librados: init done 2024-03-29T11:28:03.689 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.694+0000 7f1c5ffff640 4 set_mon_vals no callback set 2024-03-29T11:28:03.689 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.695+0000 7f1c702d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:03.689 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.695+0000 7f1c702d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:03.689 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.695+0000 7f1c702d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:28:03.689 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.695+0000 7f1c702d3640 1 -- 172.21.15.17:0/3415179562 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c50005080 con 0x7f1c680b0860 2024-03-29T11:28:03.690 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.696+0000 7f1c5effd640 1 -- 172.21.15.17:0/3415179562 <== mon.2 v1:172.21.15.17:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f1c58003c20 con 0x7f1c680b0860 2024-03-29T11:28:03.691 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.697+0000 7f1c5effd640 1 -- 172.21.15.17:0/3415179562 <== mon.2 v1:172.21.15.17:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f1c58084b90 con 0x7f1c680b0860 2024-03-29T11:28:03.691 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.697+0000 7f1c5effd640 10 client.4604.objecter ms_dispatch 0x7f1c68000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:03.691 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.697+0000 7f1c5effd640 3 client.4604.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:03.691 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.697+0000 7f1c5effd640 3 client.4604.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:03.691 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.697+0000 7f1c5effd640 20 client.4604.objecter dump_active .. 0 homeless 2024-03-29T11:28:03.691 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.697+0000 7f1c6e06f640 10 monclient: get_auth_request con 0x7f1c4c06dba0 auth_method 0 2024-03-29T11:28:03.691 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.698+0000 7f1c5effd640 10 client.4604.objecter ms_handle_connect 0x7f1c4c06dba0 2024-03-29T11:28:03.695 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.705+0000 7f1c5effd640 1 -- 172.21.15.17:0/3415179562 <== mon.2 v1:172.21.15.17:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1c58050470 con 0x7f1c680b0860 2024-03-29T11:28:03.695 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.705+0000 7f1c5effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:03.695 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:03.705+0000 7f1c5effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:04.028 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:04.039+0000 7f1c702d3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:04.028 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:04.039+0000 7f1c702d3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:04.029 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:04.039+0000 7f1c702d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:28:04.029 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:04.039+0000 7f1c702d3640 1 -- 172.21.15.17:0/3415179562 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c50005330 con 0x7f1c680b0860 2024-03-29T11:28:04.029 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:04.040+0000 7f1c5effd640 1 -- 172.21.15.17:0/3415179562 <== mon.2 v1:172.21.15.17:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f1c580523a0 con 0x7f1c680b0860 2024-03-29T11:28:04.029 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:04.040+0000 7f1c5effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:04.029 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:04.040+0000 7f1c5effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:04.030 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:28:04.030 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:04.032 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:04.043+0000 7f1c702d3640 10 librados: watch_flush enter 2024-03-29T11:28:04.034 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:04.045+0000 7f1c702d3640 10 librados: watch_flush exit 2024-03-29T11:28:04.034 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:04.045+0000 7f1c702d3640 20 client.4604.objecter shutdown clearing up homeless session... 2024-03-29T11:28:04.034 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:04.045+0000 7f1c702d3640 10 client.4604.objecter successfully canceled tick 2024-03-29T11:28:04.035 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:04.045+0000 7f1c702d3640 1 -- 172.21.15.17:0/3415179562 >> v1:172.21.15.17:6816/3236185052 conn(0x7f1c4c06dba0 legacy=0x7f1c4c070060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:04.035 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:04.045+0000 7f1c702d3640 10 monclient: shutdown 2024-03-29T11:28:04.035 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:04.045+0000 7f1c702d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:04.035 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:04.045+0000 7f1c702d3640 1 -- 172.21.15.17:0/3415179562 >> v1:172.21.15.17:6790/0 conn(0x7f1c680b0860 legacy=0x7f1c68152fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:04.035 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:04.046+0000 7f1c702d3640 1 -- 172.21.15.17:0/3415179562 shutdown_connections 2024-03-29T11:28:04.035 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:04.046+0000 7f1c702d3640 1 -- 172.21.15.17:0/3415179562 >> 172.21.15.17:0/3415179562 conn(0x7f1c6806b660 msgr2=0x7f1c680ae870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:04.036 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:04.047+0000 7f1c702d3640 1 -- 172.21.15.17:0/3415179562 shutdown_connections 2024-03-29T11:28:04.036 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:04.047+0000 7f1c702d3640 1 -- 172.21.15.17:0/3415179562 wait complete. 2024-03-29T11:28:04.036 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:04.047+0000 7f1c702d3640 1 librados: shutdown 2024-03-29T11:28:04.053 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:10.860 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:11.009 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.020+0000 7fed0d50e640 10 monclient: get_monmap_and_config 2024-03-29T11:28:11.009 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.020+0000 7fed0d50e640 10 monclient: build_initial_monmap 2024-03-29T11:28:11.010 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.020+0000 7fed0d50e640 10 monclient: monmap: 2024-03-29T11:28:11.010 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:28:11.010 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:28:11.010 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:28:11.021213+0000 2024-03-29T11:28:11.010 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:28:11.021213+0000 2024-03-29T11:28:11.010 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:11.010 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:11.010 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:28:11.010 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:28:11.010 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:28:11.010 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:11.011 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.021+0000 7fed0d50e640 1 Processor -- start 2024-03-29T11:28:11.011 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.021+0000 7fed0d50e640 1 -- start start 2024-03-29T11:28:11.011 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.021+0000 7fed0d50e640 10 monclient: init 2024-03-29T11:28:11.012 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.022+0000 7fed0d50e640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:11.012 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.022+0000 7fed0d50e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:28:11.012 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.022+0000 7fed0d50e640 10 monclient(hunting): picked mon.noname-a con 0x7fed08071c30 addr v1:172.21.15.17:6789/0 2024-03-29T11:28:11.012 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.022+0000 7fed0d50e640 10 monclient(hunting): picked mon.noname-b con 0x7fed080a96c0 addr v1:172.21.15.17:6790/0 2024-03-29T11:28:11.013 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.022+0000 7fed0d50e640 10 monclient(hunting): picked mon.noname-c con 0x7fed0813e1e0 addr v1:172.21.15.94:6789/0 2024-03-29T11:28:11.013 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.022+0000 7fed0d50e640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed080ab300 con 0x7fed08071c30 2024-03-29T11:28:11.013 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.022+0000 7fed0d50e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed080a9b00 con 0x7fed0813e1e0 2024-03-29T11:28:11.013 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.022+0000 7fed0d50e640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed080ac2f0 con 0x7fed080a96c0 2024-03-29T11:28:11.013 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.022+0000 7fed0d50e640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:11.013 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.022+0000 7fed0d50e640 10 monclient(hunting): authenticate will time out at 1397.227034s 2024-03-29T11:28:11.013 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.024+0000 7fed067fc640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7fed080a96c0 0x7fed0813dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:39168/0 (socket says 172.21.15.17:39168) 2024-03-29T11:28:11.013 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.024+0000 7fed067fc640 1 -- 172.21.15.17:0/658988874 learned_addr learned my addr 172.21.15.17:0/658988874 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:28:11.014 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.024+0000 7fed05ffb640 1 -- 172.21.15.17:0/658988874 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1629279160 0 0) 0x7fed080ac2f0 con 0x7fed080a96c0 2024-03-29T11:28:11.014 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.025+0000 7fed05ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:11.014 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.025+0000 7fed05ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:11.014 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.025+0000 7fed05ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:11.014 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.025+0000 7fed05ffb640 10 monclient(hunting): my global_id is 4607 2024-03-29T11:28:11.014 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.025+0000 7fed05ffb640 1 -- 172.21.15.17:0/658988874 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecf0003520 con 0x7fed080a96c0 2024-03-29T11:28:11.015 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.025+0000 7fed05ffb640 1 -- 172.21.15.17:0/658988874 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3043953900 0 0) 0x7fecf0003520 con 0x7fed080a96c0 2024-03-29T11:28:11.015 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.025+0000 7fed05ffb640 1 -- 172.21.15.17:0/658988874 >> v1:172.21.15.94:6789/0 conn(0x7fed0813e1e0 legacy=0x7fed080ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:11.015 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.025+0000 7fed05ffb640 1 -- 172.21.15.17:0/658988874 >> v1:172.21.15.17:6789/0 conn(0x7fed08071c30 legacy=0x7fed080b0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:11.015 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.025+0000 7fed05ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:28:11.015 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.025+0000 7fed05ffb640 1 monclient: found mon.noname-b 2024-03-29T11:28:11.015 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.025+0000 7fed05ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:11.015 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.025+0000 7fed05ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.17:6790/0 2024-03-29T11:28:11.015 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.025+0000 7fed05ffb640 1 -- 172.21.15.17:0/658988874 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed08145ec0 con 0x7fed080a96c0 2024-03-29T11:28:11.015 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.025+0000 7fed05ffb640 10 monclient: _finish_auth 0 2024-03-29T11:28:11.015 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.025+0000 7fed05ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:28:11.015 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.025+0000 7fed05ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:11.015 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.025+0000 7fed05ffb640 1 -- 172.21.15.17:0/658988874 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fecfc001690 con 0x7fed080a96c0 2024-03-29T11:28:11.015 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.025+0000 7fed05ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:11.015 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.025+0000 7fed05ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:11.016 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.025+0000 7fed05ffb640 10 monclient: dump: 2024-03-29T11:28:11.016 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:28:11.016 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:28:11.016 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:11.016 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:11.016 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:11.016 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:11.016 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:28:11.016 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:28:11.016 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:28:11.016 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:11.017 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.026+0000 7fed0d50e640 5 monclient: authenticate success, global_id 4607 2024-03-29T11:28:11.017 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.026+0000 7fed0d50e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:11.017 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.028+0000 7fed05ffb640 1 -- 172.21.15.17:0/658988874 <== mon.2 v1:172.21.15.17:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fecfc001860 con 0x7fed080a96c0 2024-03-29T11:28:11.017 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.028+0000 7fed05ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:11.018 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.028+0000 7fed0d50e640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:11.018 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.028+0000 7fed0d50e640 4 set_mon_vals no callback set 2024-03-29T11:28:11.018 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.029+0000 7fed0d50e640 10 monclient: shutdown 2024-03-29T11:28:11.018 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.029+0000 7fed0d50e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:11.018 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.029+0000 7fed0d50e640 1 -- 172.21.15.17:0/658988874 >> v1:172.21.15.17:6790/0 conn(0x7fed080a96c0 legacy=0x7fed0813dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:11.020 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.030+0000 7fed0d50e640 1 -- 172.21.15.17:0/658988874 shutdown_connections 2024-03-29T11:28:11.020 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.030+0000 7fed0d50e640 1 -- 172.21.15.17:0/658988874 >> 172.21.15.17:0/658988874 conn(0x7fed0806b660 msgr2=0x7fed0806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:11.020 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.031+0000 7fed0d50e640 1 -- 172.21.15.17:0/658988874 shutdown_connections 2024-03-29T11:28:11.020 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.031+0000 7fed0d50e640 1 -- 172.21.15.17:0/658988874 wait complete. 2024-03-29T11:28:11.021 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.032+0000 7fed0d50e640 10 monclient: build_initial_monmap 2024-03-29T11:28:11.021 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.032+0000 7fed0d50e640 10 monclient: monmap: 2024-03-29T11:28:11.021 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:28:11.021 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:11.021 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:28:11.021 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:28:11.021 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:11.022 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:11.022 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:28:11.022 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:28:11.022 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:28:11.022 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:11.022 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.032+0000 7fed0d50e640 1 librados: starting msgr at 2024-03-29T11:28:11.023 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.032+0000 7fed0d50e640 1 librados: starting objecter 2024-03-29T11:28:11.025 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.032+0000 7fed0d50e640 1 Processor -- start 2024-03-29T11:28:11.025 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.032+0000 7fed0d50e640 1 -- start start 2024-03-29T11:28:11.025 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.032+0000 7fed0d50e640 1 librados: setting wanted keys 2024-03-29T11:28:11.025 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.032+0000 7fed0d50e640 1 librados: calling monclient init 2024-03-29T11:28:11.025 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.032+0000 7fed0d50e640 10 monclient: init 2024-03-29T11:28:11.025 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.033+0000 7fed0d50e640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:11.025 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.033+0000 7fed0d50e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:28:11.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.033+0000 7fed0d50e640 10 monclient(hunting): picked mon.noname-c con 0x7fed08071c30 addr v1:172.21.15.17:6790/0 2024-03-29T11:28:11.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.033+0000 7fed0d50e640 10 monclient(hunting): picked mon.noname-a con 0x7fed080a96c0 addr v1:172.21.15.17:6789/0 2024-03-29T11:28:11.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.033+0000 7fed0d50e640 10 monclient(hunting): picked mon.noname-b con 0x7fed0813e1e0 addr v1:172.21.15.94:6789/0 2024-03-29T11:28:11.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.033+0000 7fed0d50e640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed08070110 con 0x7fed080a96c0 2024-03-29T11:28:11.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.033+0000 7fed0d50e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed0814fa20 con 0x7fed0813e1e0 2024-03-29T11:28:11.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.033+0000 7fed0d50e640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed0814e800 con 0x7fed08071c30 2024-03-29T11:28:11.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.033+0000 7fed0d50e640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:11.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.033+0000 7fed0d50e640 10 monclient(hunting): authenticate will time out at 1397.238575s 2024-03-29T11:28:11.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.033+0000 7fed06ffd640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7fed08071c30 0x7fed081513f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:39690/0 (socket says 172.21.15.17:39690) 2024-03-29T11:28:11.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.033+0000 7fed06ffd640 1 -- 172.21.15.17:0/1468203539 learned_addr learned my addr 172.21.15.17:0/1468203539 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:28:11.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.035+0000 7fecef7fe640 10 client.?.objecter ms_handle_connect 0x7fed08071c30 2024-03-29T11:28:11.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.035+0000 7fecef7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:11.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.035+0000 7fecef7fe640 1 -- 172.21.15.17:0/1468203539 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1930929355 0 0) 0x7fed0814e800 con 0x7fed08071c30 2024-03-29T11:28:11.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.035+0000 7fecef7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:28:11.026 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.035+0000 7fecef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:11.027 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.035+0000 7fecef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:11.027 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.035+0000 7fecef7fe640 10 monclient(hunting): my global_id is 4610 2024-03-29T11:28:11.027 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.035+0000 7fecef7fe640 1 -- 172.21.15.17:0/1468203539 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecf4004eb0 con 0x7fed08071c30 2024-03-29T11:28:11.031 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.037+0000 7fecef7fe640 1 -- 172.21.15.17:0/1468203539 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 587030390 0 0) 0x7fecf4004eb0 con 0x7fed08071c30 2024-03-29T11:28:11.031 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.037+0000 7fecef7fe640 1 -- 172.21.15.17:0/1468203539 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed0814e800 con 0x7fed08071c30 2024-03-29T11:28:11.031 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.037+0000 7fecef7fe640 1 -- 172.21.15.17:0/1468203539 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fecf8001960 con 0x7fed08071c30 2024-03-29T11:28:11.031 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.037+0000 7fecef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:11.031 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.037+0000 7fecef7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:11.031 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.037+0000 7fecef7fe640 10 monclient(hunting): dump: 2024-03-29T11:28:11.031 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:28:11.032 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:28:11.032 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:11.032 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:11.032 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:11.032 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:11.032 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:28:11.032 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:28:11.032 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:28:11.032 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:11.032 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.037+0000 7fecef7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:11.032 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.038+0000 7fecef7fe640 1 -- 172.21.15.17:0/1468203539 <== mon.2 v1:172.21.15.17:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2943751140 0 0) 0x7fed0814e800 con 0x7fed08071c30 2024-03-29T11:28:11.032 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.038+0000 7fecef7fe640 1 -- 172.21.15.17:0/1468203539 >> v1:172.21.15.94:6789/0 conn(0x7fed0813e1e0 legacy=0x7fed081dc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:11.033 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.038+0000 7fecef7fe640 1 -- 172.21.15.17:0/1468203539 >> v1:172.21.15.17:6789/0 conn(0x7fed080a96c0 legacy=0x7fed081db910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:11.033 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.038+0000 7fecef7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:28:11.033 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.038+0000 7fecef7fe640 1 monclient: found mon.c 2024-03-29T11:28:11.033 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.038+0000 7fecef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:11.033 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.038+0000 7fecef7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:28:11.033 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.038+0000 7fecef7fe640 1 -- 172.21.15.17:0/1468203539 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed08071110 con 0x7fed08071c30 2024-03-29T11:28:11.033 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.038+0000 7fecef7fe640 10 monclient: _finish_auth 0 2024-03-29T11:28:11.033 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.038+0000 7fecef7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:28:11.033 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.038+0000 7fecef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:11.033 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.038+0000 7fed0d50e640 5 monclient: authenticate success, global_id 4610 2024-03-29T11:28:11.033 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.038+0000 7fed0d50e640 10 monclient: _renew_subs 2024-03-29T11:28:11.033 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.038+0000 7fed0d50e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:28:11.033 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.038+0000 7fed0d50e640 1 -- 172.21.15.17:0/1468203539 --> v1:172.21.15.17:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed08151b00 con 0x7fed08071c30 2024-03-29T11:28:11.033 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.038+0000 7fed0d50e640 10 client.4610.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:11.033 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.038+0000 7fed0d50e640 10 monclient: _renew_subs 2024-03-29T11:28:11.034 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.038+0000 7fed0d50e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:28:11.034 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.038+0000 7fed0d50e640 1 -- 172.21.15.17:0/1468203539 --> v1:172.21.15.17:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fed0814e120 con 0x7fed08071c30 2024-03-29T11:28:11.034 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.038+0000 7fed0d50e640 1 librados: init done 2024-03-29T11:28:11.034 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.038+0000 7fecef7fe640 1 -- 172.21.15.17:0/1468203539 <== mon.2 v1:172.21.15.17:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fecf8001ac0 con 0x7fed08071c30 2024-03-29T11:28:11.034 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.038+0000 7fecef7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:11.034 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.038+0000 7fecef7fe640 1 -- 172.21.15.17:0/1468203539 <== mon.2 v1:172.21.15.17:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fecf8003a40 con 0x7fed08071c30 2024-03-29T11:28:11.034 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.038+0000 7fecef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:11.034 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.038+0000 7fecef7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:11.034 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.038+0000 7fecef7fe640 10 monclient: dump: 2024-03-29T11:28:11.034 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:28:11.034 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:28:11.034 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:11.034 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:11.034 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:11.035 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:11.035 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:28:11.035 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:28:11.035 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:28:11.035 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:11.035 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.040+0000 7fed0d50e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:11.035 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.040+0000 7fed0d50e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:11.035 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.040+0000 7fed0d50e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:28:11.035 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.040+0000 7fed0d50e640 1 -- 172.21.15.17:0/1468203539 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed080abb00 con 0x7fed08071c30 2024-03-29T11:28:11.035 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.040+0000 7fecef7fe640 1 -- 172.21.15.17:0/1468203539 <== mon.2 v1:172.21.15.17:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7fecf8003ba0 con 0x7fed08071c30 2024-03-29T11:28:11.035 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.041+0000 7fecef7fe640 1 -- 172.21.15.17:0/1468203539 <== mon.2 v1:172.21.15.17:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7fecf8084ae0 con 0x7fed08071c30 2024-03-29T11:28:11.035 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.041+0000 7fecef7fe640 10 client.4610.objecter ms_dispatch 0x7fed08000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:11.035 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.041+0000 7fecef7fe640 3 client.4610.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:11.035 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.041+0000 7fecef7fe640 3 client.4610.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:11.035 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.041+0000 7fecef7fe640 20 client.4610.objecter dump_active .. 0 homeless 2024-03-29T11:28:11.036 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.041+0000 7fed04ff9640 4 set_mon_vals no callback set 2024-03-29T11:28:11.036 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.041+0000 7fed067fc640 10 monclient: get_auth_request con 0x7fecf406dc50 auth_method 0 2024-03-29T11:28:11.036 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.041+0000 7fecef7fe640 10 client.4610.objecter ms_handle_connect 0x7fecf406dc50 2024-03-29T11:28:11.038 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.048+0000 7fecef7fe640 1 -- 172.21.15.17:0/1468203539 <== mon.2 v1:172.21.15.17:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fecf8050420 con 0x7fed08071c30 2024-03-29T11:28:11.038 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.048+0000 7fecef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:11.038 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.048+0000 7fecef7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:11.383 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.394+0000 7fed0d50e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:11.383 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.394+0000 7fed0d50e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:11.383 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.394+0000 7fed0d50e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:28:11.383 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.394+0000 7fed0d50e640 1 -- 172.21.15.17:0/1468203539 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fed0814e990 con 0x7fed08071c30 2024-03-29T11:28:11.384 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.395+0000 7fecef7fe640 1 -- 172.21.15.17:0/1468203539 <== mon.2 v1:172.21.15.17:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7fecf8050420 con 0x7fed08071c30 2024-03-29T11:28:11.384 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.395+0000 7fecef7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:11.384 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.395+0000 7fecef7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:11.385 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:28:11.386 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:11.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.399+0000 7fed0d50e640 10 librados: watch_flush enter 2024-03-29T11:28:11.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.399+0000 7fed0d50e640 10 librados: watch_flush exit 2024-03-29T11:28:11.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.399+0000 7fed0d50e640 20 client.4610.objecter shutdown clearing up homeless session... 2024-03-29T11:28:11.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.399+0000 7fed0d50e640 10 client.4610.objecter successfully canceled tick 2024-03-29T11:28:11.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.400+0000 7fed0d50e640 1 -- 172.21.15.17:0/1468203539 >> v1:172.21.15.17:6816/3236185052 conn(0x7fecf406dc50 legacy=0x7fecf4070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:11.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.400+0000 7fed0d50e640 10 monclient: shutdown 2024-03-29T11:28:11.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.400+0000 7fed0d50e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:11.389 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.400+0000 7fed0d50e640 1 -- 172.21.15.17:0/1468203539 >> v1:172.21.15.17:6790/0 conn(0x7fed08071c30 legacy=0x7fed081513f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:11.391 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.401+0000 7fed0d50e640 1 -- 172.21.15.17:0/1468203539 shutdown_connections 2024-03-29T11:28:11.391 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.401+0000 7fed0d50e640 1 -- 172.21.15.17:0/1468203539 >> 172.21.15.17:0/1468203539 conn(0x7fed0806b660 msgr2=0x7fed0813d8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:11.391 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.401+0000 7fed0d50e640 1 -- 172.21.15.17:0/1468203539 shutdown_connections 2024-03-29T11:28:11.391 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.401+0000 7fed0d50e640 1 -- 172.21.15.17:0/1468203539 wait complete. 2024-03-29T11:28:11.391 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:11.402+0000 7fed0d50e640 1 librados: shutdown 2024-03-29T11:28:11.408 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:18.216 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:18.385 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.392+0000 7f5f47a06640 10 monclient: get_monmap_and_config 2024-03-29T11:28:18.385 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.392+0000 7f5f47a06640 10 monclient: build_initial_monmap 2024-03-29T11:28:18.385 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.392+0000 7f5f47a06640 10 monclient: monmap: 2024-03-29T11:28:18.385 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:28:18.385 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:28:18.385 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:28:18.393847+0000 2024-03-29T11:28:18.385 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:28:18.393847+0000 2024-03-29T11:28:18.385 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:18.385 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:18.385 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:28:18.385 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:28:18.385 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:28:18.386 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:18.386 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.396+0000 7f5f47a06640 1 Processor -- start 2024-03-29T11:28:18.386 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.396+0000 7f5f47a06640 1 -- start start 2024-03-29T11:28:18.386 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.396+0000 7f5f47a06640 10 monclient: init 2024-03-29T11:28:18.387 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.397+0000 7f5f47a06640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:18.387 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.397+0000 7f5f47a06640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:28:18.387 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.397+0000 7f5f47a06640 10 monclient(hunting): picked mon.noname-b con 0x7f5f400a9930 addr v1:172.21.15.17:6790/0 2024-03-29T11:28:18.387 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.397+0000 7f5f47a06640 10 monclient(hunting): picked mon.noname-c con 0x7f5f40149dc0 addr v1:172.21.15.94:6789/0 2024-03-29T11:28:18.387 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.397+0000 7f5f47a06640 10 monclient(hunting): picked mon.noname-a con 0x7f5f4014ff40 addr v1:172.21.15.17:6789/0 2024-03-29T11:28:18.387 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.397+0000 7f5f47a06640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f400aae60 con 0x7f5f4014ff40 2024-03-29T11:28:18.387 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.397+0000 7f5f47a06640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f400b4310 con 0x7f5f40149dc0 2024-03-29T11:28:18.387 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.397+0000 7f5f47a06640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f400afdd0 con 0x7f5f400a9930 2024-03-29T11:28:18.387 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.397+0000 7f5f47a06640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:18.387 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.397+0000 7f5f47a06640 10 monclient(hunting): authenticate will time out at 1404.602608s 2024-03-29T11:28:18.390 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.400+0000 7f5f457a2640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f5f400a9930 0x7f5f40143a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:33082/0 (socket says 172.21.15.17:33082) 2024-03-29T11:28:18.390 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.400+0000 7f5f45fa3640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f5f4014ff40 0x7f5f40152330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:55641/0 (socket says 172.21.15.17:55641) 2024-03-29T11:28:18.390 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.400+0000 7f5f45fa3640 1 -- 172.21.15.17:0/3295734356 learned_addr learned my addr 172.21.15.17:0/3295734356 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:28:18.390 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.401+0000 7f5f37fff640 1 -- 172.21.15.17:0/3295734356 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3731839834 0 0) 0x7f5f400afdd0 con 0x7f5f400a9930 2024-03-29T11:28:18.391 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.401+0000 7f5f37fff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:18.391 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.401+0000 7f5f37fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:18.391 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.401+0000 7f5f37fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:18.391 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.401+0000 7f5f37fff640 10 monclient(hunting): my global_id is 4613 2024-03-29T11:28:18.391 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.401+0000 7f5f37fff640 1 -- 172.21.15.17:0/3295734356 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f2c003520 con 0x7f5f400a9930 2024-03-29T11:28:18.393 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.401+0000 7f5f37fff640 1 -- 172.21.15.17:0/3295734356 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3579220434 0 0) 0x7f5f400aae60 con 0x7f5f4014ff40 2024-03-29T11:28:18.393 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.402+0000 7f5f37fff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:18.393 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.402+0000 7f5f37fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:18.393 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.402+0000 7f5f37fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:18.393 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.402+0000 7f5f37fff640 10 monclient(hunting): my global_id is 4527 2024-03-29T11:28:18.393 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.402+0000 7f5f37fff640 1 -- 172.21.15.17:0/3295734356 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f400afdd0 con 0x7f5f4014ff40 2024-03-29T11:28:18.393 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.402+0000 7f5f37fff640 1 -- 172.21.15.17:0/3295734356 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2387451712 0 0) 0x7f5f2c003520 con 0x7f5f400a9930 2024-03-29T11:28:18.393 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.402+0000 7f5f37fff640 1 -- 172.21.15.17:0/3295734356 >> v1:172.21.15.94:6789/0 conn(0x7f5f40149dc0 legacy=0x7f5f4014a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:18.393 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.402+0000 7f5f37fff640 1 -- 172.21.15.17:0/3295734356 >> v1:172.21.15.17:6789/0 conn(0x7f5f4014ff40 legacy=0x7f5f40152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:18.393 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.402+0000 7f5f37fff640 10 monclient: _finish_hunting 0 2024-03-29T11:28:18.394 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.402+0000 7f5f37fff640 1 monclient: found mon.noname-b 2024-03-29T11:28:18.394 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.403+0000 7f5f37fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:18.394 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.403+0000 7f5f37fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.17:6790/0 2024-03-29T11:28:18.394 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.403+0000 7f5f37fff640 1 -- 172.21.15.17:0/3295734356 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f4014a950 con 0x7f5f400a9930 2024-03-29T11:28:18.394 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.403+0000 7f5f37fff640 10 monclient: _finish_auth 0 2024-03-29T11:28:18.394 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.403+0000 7f5f37fff640 10 monclient: _check_auth_tickets 2024-03-29T11:28:18.394 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.403+0000 7f5f37fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:18.394 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.403+0000 7f5f37fff640 1 -- 172.21.15.17:0/3295734356 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f5f30001690 con 0x7f5f400a9930 2024-03-29T11:28:18.394 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.403+0000 7f5f37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:18.394 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.403+0000 7f5f37fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:18.394 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.403+0000 7f5f37fff640 10 monclient: dump: 2024-03-29T11:28:18.394 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:28:18.394 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:28:18.394 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:18.394 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:18.395 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:18.395 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:18.395 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:28:18.395 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:28:18.395 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:28:18.395 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:18.395 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.403+0000 7f5f47a06640 5 monclient: authenticate success, global_id 4613 2024-03-29T11:28:18.395 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.403+0000 7f5f47a06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:18.395 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.406+0000 7f5f37fff640 1 -- 172.21.15.17:0/3295734356 <== mon.2 v1:172.21.15.17:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f300017f0 con 0x7f5f400a9930 2024-03-29T11:28:18.396 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.406+0000 7f5f37fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:18.396 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.406+0000 7f5f37fff640 1 -- 172.21.15.17:0/3295734356 <== mon.2 v1:172.21.15.17:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f5f30001c40 con 0x7f5f400a9930 2024-03-29T11:28:18.396 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.406+0000 7f5f37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:18.396 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.406+0000 7f5f37fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:18.396 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.406+0000 7f5f37fff640 10 monclient: dump: 2024-03-29T11:28:18.396 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:28:18.396 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:28:18.396 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:18.396 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:18.396 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:18.396 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:18.396 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:28:18.396 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:28:18.396 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:28:18.396 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:18.397 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.406+0000 7f5f47a06640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:18.397 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.406+0000 7f5f47a06640 4 set_mon_vals no callback set 2024-03-29T11:28:18.397 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.407+0000 7f5f47a06640 10 monclient: shutdown 2024-03-29T11:28:18.397 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.407+0000 7f5f47a06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:18.397 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.407+0000 7f5f47a06640 1 -- 172.21.15.17:0/3295734356 >> v1:172.21.15.17:6790/0 conn(0x7f5f400a9930 legacy=0x7f5f40143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:18.397 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.407+0000 7f5f47a06640 1 -- 172.21.15.17:0/3295734356 shutdown_connections 2024-03-29T11:28:18.397 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.407+0000 7f5f47a06640 1 -- 172.21.15.17:0/3295734356 >> 172.21.15.17:0/3295734356 conn(0x7f5f400b35e0 msgr2=0x7f5f400b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:18.398 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.408+0000 7f5f47a06640 1 -- 172.21.15.17:0/3295734356 shutdown_connections 2024-03-29T11:28:18.399 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.408+0000 7f5f47a06640 1 -- 172.21.15.17:0/3295734356 wait complete. 2024-03-29T11:28:18.399 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.409+0000 7f5f47a06640 10 monclient: build_initial_monmap 2024-03-29T11:28:18.399 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.409+0000 7f5f47a06640 10 monclient: monmap: 2024-03-29T11:28:18.399 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:28:18.399 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:18.399 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:28:18.400 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:28:18.400 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:18.400 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:18.400 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:28:18.400 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:28:18.400 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:28:18.400 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:18.400 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.410+0000 7f5f47a06640 1 librados: starting msgr at 2024-03-29T11:28:18.400 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.410+0000 7f5f47a06640 1 librados: starting objecter 2024-03-29T11:28:18.400 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.410+0000 7f5f47a06640 1 Processor -- start 2024-03-29T11:28:18.401 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.410+0000 7f5f47a06640 1 -- start start 2024-03-29T11:28:18.401 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.410+0000 7f5f47a06640 1 librados: setting wanted keys 2024-03-29T11:28:18.401 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.410+0000 7f5f47a06640 1 librados: calling monclient init 2024-03-29T11:28:18.401 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.410+0000 7f5f47a06640 10 monclient: init 2024-03-29T11:28:18.401 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.411+0000 7f5f47a06640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:18.401 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.411+0000 7f5f47a06640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:28:18.401 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.411+0000 7f5f47a06640 10 monclient(hunting): picked mon.noname-a con 0x7f5f400a9930 addr v1:172.21.15.17:6789/0 2024-03-29T11:28:18.402 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.411+0000 7f5f47a06640 10 monclient(hunting): picked mon.noname-c con 0x7f5f40149dc0 addr v1:172.21.15.17:6790/0 2024-03-29T11:28:18.402 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.411+0000 7f5f47a06640 10 monclient(hunting): picked mon.noname-b con 0x7f5f4014ff40 addr v1:172.21.15.94:6789/0 2024-03-29T11:28:18.402 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.411+0000 7f5f47a06640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f40147ce0 con 0x7f5f400a9930 2024-03-29T11:28:18.402 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.411+0000 7f5f47a06640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f40144d10 con 0x7f5f4014ff40 2024-03-29T11:28:18.402 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.411+0000 7f5f47a06640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f40145ee0 con 0x7f5f40149dc0 2024-03-29T11:28:18.402 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.411+0000 7f5f47a06640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:18.402 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.411+0000 7f5f47a06640 10 monclient(hunting): authenticate will time out at 1404.616381s 2024-03-29T11:28:18.402 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.411+0000 7f5f44fa1640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f5f40149dc0 0x7f5f40148200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:37054/0 (socket says 172.21.15.17:37054) 2024-03-29T11:28:18.402 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.411+0000 7f5f44fa1640 1 -- 172.21.15.17:0/3581582625 learned_addr learned my addr 172.21.15.17:0/3581582625 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:28:18.402 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.412+0000 7f5f35ffb640 10 client.?.objecter ms_handle_connect 0x7f5f40149dc0 2024-03-29T11:28:18.402 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.412+0000 7f5f35ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:18.404 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.415+0000 7f5f35ffb640 1 -- 172.21.15.17:0/3581582625 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2452407232 0 0) 0x7f5f40145ee0 con 0x7f5f40149dc0 2024-03-29T11:28:18.404 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.415+0000 7f5f35ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:18.404 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.415+0000 7f5f35ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:18.405 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.415+0000 7f5f35ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:18.405 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.415+0000 7f5f35ffb640 10 monclient(hunting): my global_id is 4616 2024-03-29T11:28:18.405 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.415+0000 7f5f35ffb640 1 -- 172.21.15.17:0/3581582625 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f24004eb0 con 0x7f5f40149dc0 2024-03-29T11:28:18.405 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.415+0000 7f5f35ffb640 10 client.?.objecter ms_handle_connect 0x7f5f4014ff40 2024-03-29T11:28:18.405 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.415+0000 7f5f35ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:18.405 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.415+0000 7f5f35ffb640 1 -- 172.21.15.17:0/3581582625 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 992776391 0 0) 0x7f5f40144d10 con 0x7f5f4014ff40 2024-03-29T11:28:18.405 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.415+0000 7f5f35ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:18.405 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.415+0000 7f5f35ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:18.405 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.415+0000 7f5f35ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:18.405 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.415+0000 7f5f35ffb640 10 monclient(hunting): my global_id is 4570 2024-03-29T11:28:18.405 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.415+0000 7f5f35ffb640 1 -- 172.21.15.17:0/3581582625 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f40145ee0 con 0x7f5f4014ff40 2024-03-29T11:28:18.406 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.415+0000 7f5f35ffb640 1 -- 172.21.15.17:0/3581582625 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 311797226 0 0) 0x7f5f40145ee0 con 0x7f5f4014ff40 2024-03-29T11:28:18.406 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.416+0000 7f5f35ffb640 1 -- 172.21.15.17:0/3581582625 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f24005490 con 0x7f5f4014ff40 2024-03-29T11:28:18.406 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.416+0000 7f5f35ffb640 1 -- 172.21.15.17:0/3581582625 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f5f3c001740 con 0x7f5f4014ff40 2024-03-29T11:28:18.406 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.416+0000 7f5f35ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:18.406 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.416+0000 7f5f35ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:18.406 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.416+0000 7f5f35ffb640 10 monclient(hunting): dump: 2024-03-29T11:28:18.406 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:28:18.407 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:28:18.407 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:18.407 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:18.407 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:18.407 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:18.407 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:28:18.407 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:28:18.407 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:28:18.407 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:18.407 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.416+0000 7f5f35ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:18.407 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.416+0000 7f5f35ffb640 1 -- 172.21.15.17:0/3581582625 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 107691390 0 0) 0x7f5f24004eb0 con 0x7f5f40149dc0 2024-03-29T11:28:18.407 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.416+0000 7f5f35ffb640 1 -- 172.21.15.17:0/3581582625 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f24005b60 con 0x7f5f40149dc0 2024-03-29T11:28:18.407 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.416+0000 7f5f35ffb640 1 -- 172.21.15.17:0/3581582625 <== mon.1 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 714457219 0 0) 0x7f5f24005490 con 0x7f5f4014ff40 2024-03-29T11:28:18.407 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.416+0000 7f5f35ffb640 1 -- 172.21.15.17:0/3581582625 >> v1:172.21.15.17:6790/0 conn(0x7f5f40149dc0 legacy=0x7f5f40148200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:18.407 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.416+0000 7f5f35ffb640 1 -- 172.21.15.17:0/3581582625 >> v1:172.21.15.17:6789/0 conn(0x7f5f400a9930 legacy=0x7f5f401457d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:18.408 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.416+0000 7f5f35ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:28:18.408 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.416+0000 7f5f35ffb640 1 monclient: found mon.b 2024-03-29T11:28:18.408 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.416+0000 7f5f35ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:18.408 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.416+0000 7f5f35ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:28:18.408 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.416+0000 7f5f35ffb640 1 -- 172.21.15.17:0/3581582625 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f40146070 con 0x7f5f4014ff40 2024-03-29T11:28:18.408 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.416+0000 7f5f35ffb640 10 monclient: _finish_auth 0 2024-03-29T11:28:18.408 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.416+0000 7f5f35ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:28:18.408 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.416+0000 7f5f35ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:18.408 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.416+0000 7f5f47a06640 5 monclient: authenticate success, global_id 4570 2024-03-29T11:28:18.408 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.416+0000 7f5f47a06640 10 monclient: _renew_subs 2024-03-29T11:28:18.408 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.416+0000 7f5f47a06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:28:18.408 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.416+0000 7f5f47a06640 1 -- 172.21.15.17:0/3581582625 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f40146340 con 0x7f5f4014ff40 2024-03-29T11:28:18.408 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.416+0000 7f5f47a06640 10 client.4570.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:18.408 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.416+0000 7f5f47a06640 10 monclient: _renew_subs 2024-03-29T11:28:18.408 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.416+0000 7f5f47a06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:28:18.409 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.416+0000 7f5f47a06640 1 -- 172.21.15.17:0/3581582625 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f401ef910 con 0x7f5f4014ff40 2024-03-29T11:28:18.409 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.416+0000 7f5f47a06640 1 librados: init done 2024-03-29T11:28:18.409 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.417+0000 7f5f35ffb640 1 -- 172.21.15.17:0/3581582625 <== mon.1 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f3c001ca0 con 0x7f5f4014ff40 2024-03-29T11:28:18.409 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.417+0000 7f5f35ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:18.409 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.417+0000 7f5f36ffd640 4 set_mon_vals no callback set 2024-03-29T11:28:18.409 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.418+0000 7f5f35ffb640 1 -- 172.21.15.17:0/3581582625 <== mon.1 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f5f3c001740 con 0x7f5f4014ff40 2024-03-29T11:28:18.409 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.418+0000 7f5f35ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:18.410 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.418+0000 7f5f35ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:18.410 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.418+0000 7f5f35ffb640 10 monclient: dump: 2024-03-29T11:28:18.410 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:28:18.410 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:28:18.410 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:18.410 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:18.410 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:18.410 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:18.410 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:28:18.410 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:28:18.410 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:28:18.410 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:18.410 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.419+0000 7f5f35ffb640 1 -- 172.21.15.17:0/3581582625 <== mon.1 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f5f3c01a650 con 0x7f5f4014ff40 2024-03-29T11:28:18.411 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.420+0000 7f5f35ffb640 1 -- 172.21.15.17:0/3581582625 <== mon.1 v1:172.21.15.94:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f5f3c001560 con 0x7f5f4014ff40 2024-03-29T11:28:18.411 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.420+0000 7f5f35ffb640 10 client.4570.objecter ms_dispatch 0x7f5f40000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:18.411 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.420+0000 7f5f35ffb640 3 client.4570.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:18.411 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.420+0000 7f5f35ffb640 3 client.4570.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:18.411 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.420+0000 7f5f457a2640 10 monclient: get_auth_request con 0x7f5f2406e420 auth_method 0 2024-03-29T11:28:18.411 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.420+0000 7f5f35ffb640 20 client.4570.objecter dump_active .. 0 homeless 2024-03-29T11:28:18.411 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.420+0000 7f5f35ffb640 10 client.4570.objecter ms_handle_connect 0x7f5f2406e420 2024-03-29T11:28:18.413 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.422+0000 7f5f47a06640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:18.413 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.422+0000 7f5f47a06640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:18.413 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.422+0000 7f5f47a06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:28:18.413 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.422+0000 7f5f47a06640 1 -- 172.21.15.17:0/3581582625 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f20005080 con 0x7f5f4014ff40 2024-03-29T11:28:18.419 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.430+0000 7f5f35ffb640 1 -- 172.21.15.17:0/3581582625 <== mon.1 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5f3c0502f0 con 0x7f5f4014ff40 2024-03-29T11:28:18.420 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.430+0000 7f5f35ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:18.420 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.430+0000 7f5f35ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:18.773 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.783+0000 7f5f47a06640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:18.773 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.783+0000 7f5f47a06640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:18.773 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.783+0000 7f5f47a06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:28:18.773 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.783+0000 7f5f47a06640 1 -- 172.21.15.17:0/3581582625 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f20005330 con 0x7f5f4014ff40 2024-03-29T11:28:18.773 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.784+0000 7f5f35ffb640 1 -- 172.21.15.17:0/3581582625 <== mon.1 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f5f3c0502f0 con 0x7f5f4014ff40 2024-03-29T11:28:18.774 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.784+0000 7f5f35ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:18.774 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.784+0000 7f5f35ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:18.774 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:28:18.774 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:18.779 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.788+0000 7f5f47a06640 10 librados: watch_flush enter 2024-03-29T11:28:18.779 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.790+0000 7f5f47a06640 10 librados: watch_flush exit 2024-03-29T11:28:18.780 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.790+0000 7f5f47a06640 20 client.4570.objecter shutdown clearing up homeless session... 2024-03-29T11:28:18.780 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.790+0000 7f5f47a06640 10 client.4570.objecter successfully canceled tick 2024-03-29T11:28:18.780 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.790+0000 7f5f47a06640 1 -- 172.21.15.17:0/3581582625 >> v1:172.21.15.17:6816/3236185052 conn(0x7f5f2406e420 legacy=0x7f5f240708e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:18.780 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.790+0000 7f5f47a06640 10 monclient: shutdown 2024-03-29T11:28:18.780 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.790+0000 7f5f47a06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:18.780 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.790+0000 7f5f47a06640 1 -- 172.21.15.17:0/3581582625 >> v1:172.21.15.94:6789/0 conn(0x7f5f4014ff40 legacy=0x7f5f40148910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:18.781 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.791+0000 7f5f47a06640 1 -- 172.21.15.17:0/3581582625 shutdown_connections 2024-03-29T11:28:18.781 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.791+0000 7f5f47a06640 1 -- 172.21.15.17:0/3581582625 >> 172.21.15.17:0/3581582625 conn(0x7f5f400b35e0 msgr2=0x7f5f400b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:18.781 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.791+0000 7f5f47a06640 1 -- 172.21.15.17:0/3581582625 shutdown_connections 2024-03-29T11:28:18.781 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.791+0000 7f5f47a06640 1 -- 172.21.15.17:0/3581582625 wait complete. 2024-03-29T11:28:18.781 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:18.791+0000 7f5f47a06640 1 librados: shutdown 2024-03-29T11:28:18.797 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:25.606 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:25.771 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.781+0000 7f573ed15640 10 monclient: get_monmap_and_config 2024-03-29T11:28:25.771 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.781+0000 7f573ed15640 10 monclient: build_initial_monmap 2024-03-29T11:28:25.771 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.781+0000 7f573ed15640 10 monclient: monmap: 2024-03-29T11:28:25.771 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:28:25.771 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:28:25.771 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:28:25.782284+0000 2024-03-29T11:28:25.771 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:28:25.782284+0000 2024-03-29T11:28:25.771 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:25.771 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:25.771 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:28:25.771 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:28:25.771 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:28:25.772 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:25.772 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.782+0000 7f573ed15640 1 Processor -- start 2024-03-29T11:28:25.772 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.782+0000 7f573ed15640 1 -- start start 2024-03-29T11:28:25.772 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.782+0000 7f573ed15640 10 monclient: init 2024-03-29T11:28:25.773 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.783+0000 7f573ed15640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:25.773 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.783+0000 7f573ed15640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:28:25.773 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.783+0000 7f573ed15640 10 monclient(hunting): picked mon.noname-c con 0x7f57380b0770 addr v1:172.21.15.94:6789/0 2024-03-29T11:28:25.773 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.783+0000 7f573ed15640 10 monclient(hunting): picked mon.noname-a con 0x7f573814a700 addr v1:172.21.15.17:6789/0 2024-03-29T11:28:25.773 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.783+0000 7f573ed15640 10 monclient(hunting): picked mon.noname-b con 0x7f57380ac5a0 addr v1:172.21.15.17:6790/0 2024-03-29T11:28:25.773 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.783+0000 7f573ed15640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57380aba70 con 0x7f573814a700 2024-03-29T11:28:25.773 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.783+0000 7f573ed15640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57380aecb0 con 0x7f57380b0770 2024-03-29T11:28:25.773 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.783+0000 7f573ed15640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57380af150 con 0x7f57380ac5a0 2024-03-29T11:28:25.774 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.783+0000 7f573ed15640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:25.774 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.783+0000 7f573ed15640 10 monclient(hunting): authenticate will time out at 1411.988643s 2024-03-29T11:28:25.774 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.783+0000 7f573cab1640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f57380b0770 0x7f5738149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:37362/0 (socket says 172.21.15.17:37362) 2024-03-29T11:28:25.774 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.783+0000 7f573cab1640 1 -- 172.21.15.17:0/2661310271 learned_addr learned my addr 172.21.15.17:0/2661310271 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:28:25.774 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.784+0000 7f57377fe640 1 -- 172.21.15.17:0/2661310271 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1528777951 0 0) 0x7f57380aecb0 con 0x7f57380b0770 2024-03-29T11:28:25.774 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.784+0000 7f57377fe640 10 monclient(hunting): _negotiate 2024-03-29T11:28:25.774 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.784+0000 7f57377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:25.774 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.784+0000 7f57377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:25.774 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.784+0000 7f57377fe640 10 monclient(hunting): my global_id is 4573 2024-03-29T11:28:25.775 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.784+0000 7f57377fe640 1 -- 172.21.15.17:0/2661310271 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5720003520 con 0x7f57380b0770 2024-03-29T11:28:25.775 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.785+0000 7f57377fe640 1 -- 172.21.15.17:0/2661310271 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3207002952 0 0) 0x7f57380aba70 con 0x7f573814a700 2024-03-29T11:28:25.775 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.785+0000 7f57377fe640 10 monclient(hunting): _negotiate 2024-03-29T11:28:25.775 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.785+0000 7f57377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:25.775 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.785+0000 7f57377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:25.775 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.785+0000 7f57377fe640 10 monclient(hunting): my global_id is 4530 2024-03-29T11:28:25.775 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.785+0000 7f57377fe640 1 -- 172.21.15.17:0/2661310271 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57380aecb0 con 0x7f573814a700 2024-03-29T11:28:25.775 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.785+0000 7f57377fe640 1 -- 172.21.15.17:0/2661310271 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4103501432 0 0) 0x7f57380aecb0 con 0x7f573814a700 2024-03-29T11:28:25.775 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.785+0000 7f57377fe640 1 -- 172.21.15.17:0/2661310271 >> v1:172.21.15.17:6790/0 conn(0x7f57380ac5a0 legacy=0x7f5738151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:25.776 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.785+0000 7f57377fe640 1 -- 172.21.15.17:0/2661310271 >> v1:172.21.15.94:6789/0 conn(0x7f57380b0770 legacy=0x7f5738149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:25.776 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.785+0000 7f57377fe640 10 monclient: _finish_hunting 0 2024-03-29T11:28:25.776 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.785+0000 7f57377fe640 1 monclient: found mon.noname-a 2024-03-29T11:28:25.776 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.785+0000 7f57377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:25.776 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.785+0000 7f57377fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.17:6789/0 2024-03-29T11:28:25.776 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.785+0000 7f57377fe640 1 -- 172.21.15.17:0/2661310271 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57380ac9d0 con 0x7f573814a700 2024-03-29T11:28:25.776 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.785+0000 7f57377fe640 10 monclient: _finish_auth 0 2024-03-29T11:28:25.776 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.785+0000 7f57377fe640 10 monclient: _check_auth_tickets 2024-03-29T11:28:25.776 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.785+0000 7f57377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:25.776 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.785+0000 7f57377fe640 1 -- 172.21.15.17:0/2661310271 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f572c001670 con 0x7f573814a700 2024-03-29T11:28:25.776 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.785+0000 7f57377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:25.776 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.785+0000 7f57377fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:25.776 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.785+0000 7f57377fe640 10 monclient: dump: 2024-03-29T11:28:25.776 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:28:25.776 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:28:25.777 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:25.777 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:25.777 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:25.777 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:25.777 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:28:25.777 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:28:25.777 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:28:25.777 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:25.777 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.785+0000 7f573ed15640 5 monclient: authenticate success, global_id 4530 2024-03-29T11:28:25.777 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.785+0000 7f573ed15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:25.778 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.786+0000 7f57377fe640 1 -- 172.21.15.17:0/2661310271 <== mon.0 v1:172.21.15.17:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f572c0017d0 con 0x7f573814a700 2024-03-29T11:28:25.778 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.786+0000 7f57377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:25.778 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.786+0000 7f57377fe640 1 -- 172.21.15.17:0/2661310271 <== mon.0 v1:172.21.15.17:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f572c001be0 con 0x7f573814a700 2024-03-29T11:28:25.778 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.786+0000 7f57377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:25.779 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.786+0000 7f57377fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:25.779 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.786+0000 7f57377fe640 10 monclient: dump: 2024-03-29T11:28:25.779 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:28:25.779 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:28:25.779 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:25.779 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:25.779 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:25.779 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:25.779 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:28:25.779 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:28:25.779 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:28:25.779 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:25.779 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.786+0000 7f573ed15640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:25.779 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.786+0000 7f573ed15640 4 set_mon_vals no callback set 2024-03-29T11:28:25.779 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.787+0000 7f573ed15640 10 monclient: shutdown 2024-03-29T11:28:25.779 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.787+0000 7f573ed15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:25.780 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.787+0000 7f573ed15640 1 -- 172.21.15.17:0/2661310271 >> v1:172.21.15.17:6789/0 conn(0x7f573814a700 legacy=0x7f57380abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:25.780 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.788+0000 7f573ed15640 1 -- 172.21.15.17:0/2661310271 shutdown_connections 2024-03-29T11:28:25.780 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.788+0000 7f573ed15640 1 -- 172.21.15.17:0/2661310271 >> 172.21.15.17:0/2661310271 conn(0x7f573806b660 msgr2=0x7f573806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:25.780 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.788+0000 7f573ed15640 1 -- 172.21.15.17:0/2661310271 shutdown_connections 2024-03-29T11:28:25.780 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.788+0000 7f573ed15640 1 -- 172.21.15.17:0/2661310271 wait complete. 2024-03-29T11:28:25.780 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.789+0000 7f573ed15640 10 monclient: build_initial_monmap 2024-03-29T11:28:25.781 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.789+0000 7f573ed15640 10 monclient: monmap: 2024-03-29T11:28:25.781 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:28:25.781 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:25.781 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:28:25.781 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:28:25.781 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:25.781 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:25.781 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:28:25.781 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:28:25.781 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:28:25.781 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:25.781 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.790+0000 7f573ed15640 1 librados: starting msgr at 2024-03-29T11:28:25.781 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.790+0000 7f573ed15640 1 librados: starting objecter 2024-03-29T11:28:25.781 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.790+0000 7f573ed15640 1 Processor -- start 2024-03-29T11:28:25.784 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.790+0000 7f573ed15640 1 -- start start 2024-03-29T11:28:25.784 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.790+0000 7f573ed15640 1 librados: setting wanted keys 2024-03-29T11:28:25.784 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.790+0000 7f573ed15640 1 librados: calling monclient init 2024-03-29T11:28:25.784 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.790+0000 7f573ed15640 10 monclient: init 2024-03-29T11:28:25.784 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.791+0000 7f573ed15640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:25.784 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.791+0000 7f573ed15640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:28:25.785 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.791+0000 7f573ed15640 10 monclient(hunting): picked mon.noname-c con 0x7f57380ac5a0 addr v1:172.21.15.17:6790/0 2024-03-29T11:28:25.785 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.791+0000 7f573ed15640 10 monclient(hunting): picked mon.noname-a con 0x7f57380b0770 addr v1:172.21.15.17:6789/0 2024-03-29T11:28:25.785 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.791+0000 7f573ed15640 10 monclient(hunting): picked mon.noname-b con 0x7f573814a700 addr v1:172.21.15.94:6789/0 2024-03-29T11:28:25.785 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.791+0000 7f573ed15640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57381dfa20 con 0x7f57380b0770 2024-03-29T11:28:25.785 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.792+0000 7f573ed15640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5738127860 con 0x7f573814a700 2024-03-29T11:28:25.785 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.792+0000 7f573ed15640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57381285a0 con 0x7f57380ac5a0 2024-03-29T11:28:25.785 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.792+0000 7f573ed15640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:25.785 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.792+0000 7f573ed15640 10 monclient(hunting): authenticate will time out at 1411.996900s 2024-03-29T11:28:25.785 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.792+0000 7f5737fff640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f57380b0770 0x7f57381261b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:34768/0 (socket says 172.21.15.17:34768) 2024-03-29T11:28:25.785 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.792+0000 7f5737fff640 1 -- 172.21.15.17:0/2679910884 learned_addr learned my addr 172.21.15.17:0/2679910884 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:28:25.785 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.793+0000 7f57357fa640 10 client.?.objecter ms_handle_connect 0x7f57380b0770 2024-03-29T11:28:25.785 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.793+0000 7f57357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:25.785 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.793+0000 7f57357fa640 1 -- 172.21.15.17:0/2679910884 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1701909949 0 0) 0x7f57381dfa20 con 0x7f57380b0770 2024-03-29T11:28:25.785 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.793+0000 7f57357fa640 10 monclient(hunting): _negotiate 2024-03-29T11:28:25.786 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.793+0000 7f57357fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:25.786 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.793+0000 7f57357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:25.786 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.793+0000 7f57357fa640 10 monclient(hunting): my global_id is 4533 2024-03-29T11:28:25.786 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.793+0000 7f57357fa640 1 -- 172.21.15.17:0/2679910884 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5724004eb0 con 0x7f57380b0770 2024-03-29T11:28:25.786 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.793+0000 7f57357fa640 10 client.?.objecter ms_handle_connect 0x7f57380ac5a0 2024-03-29T11:28:25.786 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.793+0000 7f57357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:25.786 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.793+0000 7f57357fa640 10 client.?.objecter ms_handle_connect 0x7f573814a700 2024-03-29T11:28:25.786 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.793+0000 7f57357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:25.786 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.793+0000 7f57357fa640 1 -- 172.21.15.17:0/2679910884 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1108940614 0 0) 0x7f57381285a0 con 0x7f57380ac5a0 2024-03-29T11:28:25.786 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.793+0000 7f57357fa640 10 monclient(hunting): _negotiate 2024-03-29T11:28:25.786 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.793+0000 7f57357fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:25.786 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.793+0000 7f57357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:25.786 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.793+0000 7f57357fa640 10 monclient(hunting): my global_id is 4622 2024-03-29T11:28:25.786 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.793+0000 7f57357fa640 1 -- 172.21.15.17:0/2679910884 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57240051d0 con 0x7f57380ac5a0 2024-03-29T11:28:25.786 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.794+0000 7f57357fa640 1 -- 172.21.15.17:0/2679910884 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1912274317 0 0) 0x7f57240051d0 con 0x7f57380ac5a0 2024-03-29T11:28:25.787 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.794+0000 7f57357fa640 1 -- 172.21.15.17:0/2679910884 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57381285a0 con 0x7f57380ac5a0 2024-03-29T11:28:25.787 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.794+0000 7f57357fa640 1 -- 172.21.15.17:0/2679910884 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f57280018f0 con 0x7f57380ac5a0 2024-03-29T11:28:25.787 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.794+0000 7f57357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:25.787 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.794+0000 7f57357fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:25.787 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.794+0000 7f57357fa640 10 monclient(hunting): dump: 2024-03-29T11:28:25.787 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:28:25.787 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:28:25.787 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:25.787 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:25.787 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:25.787 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:25.787 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:28:25.787 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:28:25.787 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:28:25.787 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:25.788 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.794+0000 7f57357fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:25.788 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.794+0000 7f57357fa640 1 -- 172.21.15.17:0/2679910884 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2664811554 0 0) 0x7f5738127860 con 0x7f573814a700 2024-03-29T11:28:25.788 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.794+0000 7f57357fa640 10 monclient(hunting): _negotiate 2024-03-29T11:28:25.788 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.794+0000 7f57357fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:25.788 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.794+0000 7f57357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:25.788 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.794+0000 7f57357fa640 10 monclient(hunting): my global_id is 4576 2024-03-29T11:28:25.788 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.794+0000 7f57357fa640 1 -- 172.21.15.17:0/2679910884 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57240051d0 con 0x7f573814a700 2024-03-29T11:28:25.788 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.794+0000 7f57357fa640 1 -- 172.21.15.17:0/2679910884 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1861088906 0 0) 0x7f5724004eb0 con 0x7f57380b0770 2024-03-29T11:28:25.788 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.794+0000 7f57357fa640 1 -- 172.21.15.17:0/2679910884 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5738127860 con 0x7f57380b0770 2024-03-29T11:28:25.788 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.794+0000 7f57357fa640 1 -- 172.21.15.17:0/2679910884 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f572c001c20 con 0x7f57380b0770 2024-03-29T11:28:25.788 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.794+0000 7f57357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:25.788 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.794+0000 7f57357fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:25.788 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.794+0000 7f57357fa640 10 monclient(hunting): dump: 2024-03-29T11:28:25.788 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:28:25.788 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:28:25.789 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:25.789 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:25.789 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:25.789 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:25.789 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:28:25.789 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:28:25.789 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:28:25.789 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:25.789 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.794+0000 7f57357fa640 1 -- 172.21.15.17:0/2679910884 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3197675268 0 0) 0x7f57240051d0 con 0x7f573814a700 2024-03-29T11:28:25.789 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.794+0000 7f57357fa640 1 -- 172.21.15.17:0/2679910884 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5724004eb0 con 0x7f573814a700 2024-03-29T11:28:25.789 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.794+0000 7f57357fa640 1 -- 172.21.15.17:0/2679910884 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f5730001a80 con 0x7f573814a700 2024-03-29T11:28:25.789 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.794+0000 7f57357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:25.789 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.794+0000 7f57357fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:25.789 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.794+0000 7f57357fa640 10 monclient(hunting): dump: 2024-03-29T11:28:25.789 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:28:25.790 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:28:25.790 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:25.790 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:25.790 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:25.790 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:25.790 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:28:25.790 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:28:25.790 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:28:25.790 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:25.792 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.795+0000 7f57357fa640 1 -- 172.21.15.17:0/2679910884 <== mon.1 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1472104543 0 0) 0x7f5724004eb0 con 0x7f573814a700 2024-03-29T11:28:25.792 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.795+0000 7f57357fa640 1 -- 172.21.15.17:0/2679910884 >> v1:172.21.15.17:6790/0 conn(0x7f57380ac5a0 legacy=0x7f57381df170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:25.792 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.795+0000 7f57357fa640 1 -- 172.21.15.17:0/2679910884 >> v1:172.21.15.17:6789/0 conn(0x7f57380b0770 legacy=0x7f57381261b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:25.792 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.795+0000 7f57357fa640 10 monclient: _finish_hunting 0 2024-03-29T11:28:25.792 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.795+0000 7f57357fa640 1 monclient: found mon.b 2024-03-29T11:28:25.792 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.795+0000 7f57357fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:25.792 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.795+0000 7f57357fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:28:25.792 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.795+0000 7f57357fa640 1 -- 172.21.15.17:0/2679910884 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5738125630 con 0x7f573814a700 2024-03-29T11:28:25.792 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.795+0000 7f57357fa640 10 monclient: _finish_auth 0 2024-03-29T11:28:25.793 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.795+0000 7f57357fa640 10 monclient: _check_auth_tickets 2024-03-29T11:28:25.793 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.795+0000 7f57357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:25.793 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.795+0000 7f573ed15640 5 monclient: authenticate success, global_id 4576 2024-03-29T11:28:25.793 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.795+0000 7f573ed15640 10 monclient: _renew_subs 2024-03-29T11:28:25.793 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.795+0000 7f573ed15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:28:25.793 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.795+0000 7f573ed15640 1 -- 172.21.15.17:0/2679910884 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5738145010 con 0x7f573814a700 2024-03-29T11:28:25.793 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.795+0000 7f573ed15640 10 client.4576.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:25.793 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.795+0000 7f573ed15640 10 monclient: _renew_subs 2024-03-29T11:28:25.793 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.795+0000 7f573ed15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:28:25.793 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.795+0000 7f573ed15640 1 -- 172.21.15.17:0/2679910884 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f57381268c0 con 0x7f573814a700 2024-03-29T11:28:25.793 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.795+0000 7f573ed15640 1 librados: init done 2024-03-29T11:28:25.793 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.796+0000 7f57357fa640 1 -- 172.21.15.17:0/2679910884 <== mon.1 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5730001fb0 con 0x7f573814a700 2024-03-29T11:28:25.793 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.796+0000 7f57357fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:25.793 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.796+0000 7f57357fa640 1 -- 172.21.15.17:0/2679910884 <== mon.1 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f5730001a80 con 0x7f573814a700 2024-03-29T11:28:25.793 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.796+0000 7f57357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:25.794 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.796+0000 7f57357fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:25.794 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.796+0000 7f57357fa640 10 monclient: dump: 2024-03-29T11:28:25.794 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:28:25.794 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:28:25.794 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:25.794 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:25.794 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:25.794 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:25.794 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:28:25.794 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:28:25.794 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:28:25.794 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:25.794 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.796+0000 7f57367fc640 4 set_mon_vals no callback set 2024-03-29T11:28:25.794 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.796+0000 7f573ed15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:25.794 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.797+0000 7f573ed15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:25.795 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.797+0000 7f573ed15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:28:25.795 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.797+0000 7f573ed15640 1 -- 172.21.15.17:0/2679910884 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5718005080 con 0x7f573814a700 2024-03-29T11:28:25.795 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.798+0000 7f57357fa640 1 -- 172.21.15.17:0/2679910884 <== mon.1 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f57300197e0 con 0x7f573814a700 2024-03-29T11:28:25.795 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.799+0000 7f57357fa640 1 -- 172.21.15.17:0/2679910884 <== mon.1 v1:172.21.15.94:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f57300018a0 con 0x7f573814a700 2024-03-29T11:28:25.795 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.799+0000 7f57357fa640 10 client.4576.objecter ms_dispatch 0x7f5738000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:25.795 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.799+0000 7f57357fa640 3 client.4576.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:25.795 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.799+0000 7f57357fa640 3 client.4576.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:25.795 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.799+0000 7f57357fa640 20 client.4576.objecter dump_active .. 0 homeless 2024-03-29T11:28:25.795 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.800+0000 7f573cab1640 10 monclient: get_auth_request con 0x7f572406e660 auth_method 0 2024-03-29T11:28:25.795 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.800+0000 7f57357fa640 10 client.4576.objecter ms_handle_connect 0x7f572406e660 2024-03-29T11:28:25.796 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.806+0000 7f57357fa640 1 -- 172.21.15.17:0/2679910884 <== mon.1 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f573004f650 con 0x7f573814a700 2024-03-29T11:28:25.796 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.806+0000 7f57357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:25.796 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:25.806+0000 7f57357fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:26.243 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:26.253+0000 7f573ed15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:26.243 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:26.253+0000 7f573ed15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:26.243 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:26.253+0000 7f573ed15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:28:26.244 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:26.253+0000 7f573ed15640 1 -- 172.21.15.17:0/2679910884 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5718005330 con 0x7f573814a700 2024-03-29T11:28:26.244 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:26.254+0000 7f57357fa640 1 -- 172.21.15.17:0/2679910884 <== mon.1 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f573004f650 con 0x7f573814a700 2024-03-29T11:28:26.244 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:26.254+0000 7f57357fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:26.244 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:26.254+0000 7f57357fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:26.244 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:28:26.245 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:26.248 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:26.258+0000 7f573ed15640 10 librados: watch_flush enter 2024-03-29T11:28:26.248 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:26.258+0000 7f573ed15640 10 librados: watch_flush exit 2024-03-29T11:28:26.248 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:26.259+0000 7f573ed15640 20 client.4576.objecter shutdown clearing up homeless session... 2024-03-29T11:28:26.248 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:26.259+0000 7f573ed15640 10 client.4576.objecter successfully canceled tick 2024-03-29T11:28:26.249 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:26.259+0000 7f573ed15640 1 -- 172.21.15.17:0/2679910884 >> v1:172.21.15.17:6816/3236185052 conn(0x7f572406e660 legacy=0x7f5724070b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:26.249 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:26.259+0000 7f573ed15640 10 monclient: shutdown 2024-03-29T11:28:26.249 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:26.259+0000 7f573ed15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:26.249 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:26.259+0000 7f573ed15640 1 -- 172.21.15.17:0/2679910884 >> v1:172.21.15.94:6789/0 conn(0x7f573814a700 legacy=0x7f57381e6440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:26.249 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:26.259+0000 7f573ed15640 1 -- 172.21.15.17:0/2679910884 shutdown_connections 2024-03-29T11:28:26.249 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:26.259+0000 7f573ed15640 1 -- 172.21.15.17:0/2679910884 >> 172.21.15.17:0/2679910884 conn(0x7f573806b660 msgr2=0x7f57380abe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:26.250 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:26.260+0000 7f573ed15640 1 -- 172.21.15.17:0/2679910884 shutdown_connections 2024-03-29T11:28:26.250 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:26.260+0000 7f573ed15640 1 -- 172.21.15.17:0/2679910884 wait complete. 2024-03-29T11:28:26.250 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:26.260+0000 7f573ed15640 1 librados: shutdown 2024-03-29T11:28:26.267 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:33.074 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:33.207 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.217+0000 7f7d8359e640 10 monclient: get_monmap_and_config 2024-03-29T11:28:33.208 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.217+0000 7f7d8359e640 10 monclient: build_initial_monmap 2024-03-29T11:28:33.208 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.217+0000 7f7d8359e640 10 monclient: monmap: 2024-03-29T11:28:33.208 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:28:33.208 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:28:33.208 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:28:33.218696+0000 2024-03-29T11:28:33.208 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:28:33.218696+0000 2024-03-29T11:28:33.208 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:33.208 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:33.208 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:28:33.208 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:28:33.208 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:28:33.208 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:33.210 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.218+0000 7f7d8359e640 1 Processor -- start 2024-03-29T11:28:33.210 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.218+0000 7f7d8359e640 1 -- start start 2024-03-29T11:28:33.210 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.218+0000 7f7d8359e640 10 monclient: init 2024-03-29T11:28:33.210 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.219+0000 7f7d8359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:33.210 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.219+0000 7f7d8359e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:28:33.210 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.219+0000 7f7d8359e640 10 monclient(hunting): picked mon.noname-c con 0x7f7d840b2800 addr v1:172.21.15.94:6789/0 2024-03-29T11:28:33.210 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.219+0000 7f7d8359e640 10 monclient(hunting): picked mon.noname-b con 0x7f7d840a9c00 addr v1:172.21.15.17:6790/0 2024-03-29T11:28:33.210 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.219+0000 7f7d8359e640 10 monclient(hunting): picked mon.noname-a con 0x7f7d840b38c0 addr v1:172.21.15.17:6789/0 2024-03-29T11:28:33.210 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.219+0000 7f7d8359e640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d84141eb0 con 0x7f7d840b38c0 2024-03-29T11:28:33.210 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.219+0000 7f7d8359e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d840b22d0 con 0x7f7d840b2800 2024-03-29T11:28:33.210 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.219+0000 7f7d8359e640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d84142d30 con 0x7f7d840a9c00 2024-03-29T11:28:33.210 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.219+0000 7f7d8359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:33.211 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.219+0000 7f7d8359e640 10 monclient(hunting): authenticate will time out at 1419.424661s 2024-03-29T11:28:33.211 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.219+0000 7f7d82d9d640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f7d840b38c0 0x7f7d8414d7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:37354/0 (socket says 172.21.15.17:37354) 2024-03-29T11:28:33.211 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.219+0000 7f7d82d9d640 1 -- 172.21.15.17:0/2664991829 learned_addr learned my addr 172.21.15.17:0/2664991829 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:28:33.211 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.220+0000 7f7d8159a640 1 -- 172.21.15.17:0/2664991829 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1406831968 0 0) 0x7f7d84141eb0 con 0x7f7d840b38c0 2024-03-29T11:28:33.211 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.220+0000 7f7d8159a640 10 monclient(hunting): _negotiate 2024-03-29T11:28:33.211 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.220+0000 7f7d8159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:33.211 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.220+0000 7f7d8159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:33.211 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.220+0000 7f7d8159a640 10 monclient(hunting): my global_id is 4536 2024-03-29T11:28:33.211 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.220+0000 7f7d8159a640 1 -- 172.21.15.17:0/2664991829 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d70003520 con 0x7f7d840b38c0 2024-03-29T11:28:33.213 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.220+0000 7f7d8159a640 1 -- 172.21.15.17:0/2664991829 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1750523085 0 0) 0x7f7d84142d30 con 0x7f7d840a9c00 2024-03-29T11:28:33.213 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.220+0000 7f7d8159a640 10 monclient(hunting): _negotiate 2024-03-29T11:28:33.213 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.220+0000 7f7d8159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:33.213 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.220+0000 7f7d8159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:33.214 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.220+0000 7f7d8159a640 10 monclient(hunting): my global_id is 4625 2024-03-29T11:28:33.214 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.220+0000 7f7d8159a640 1 -- 172.21.15.17:0/2664991829 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d84141eb0 con 0x7f7d840a9c00 2024-03-29T11:28:33.214 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.221+0000 7f7d8159a640 1 -- 172.21.15.17:0/2664991829 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2126690528 0 0) 0x7f7d70003520 con 0x7f7d840b38c0 2024-03-29T11:28:33.214 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.221+0000 7f7d8159a640 1 -- 172.21.15.17:0/2664991829 >> v1:172.21.15.17:6790/0 conn(0x7f7d840a9c00 legacy=0x7f7d8413bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:33.214 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.221+0000 7f7d8159a640 1 -- 172.21.15.17:0/2664991829 >> v1:172.21.15.94:6789/0 conn(0x7f7d840b2800 legacy=0x7f7d84142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:33.214 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.221+0000 7f7d8159a640 10 monclient: _finish_hunting 0 2024-03-29T11:28:33.214 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.221+0000 7f7d8159a640 1 monclient: found mon.noname-a 2024-03-29T11:28:33.214 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.221+0000 7f7d8159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:33.214 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.221+0000 7f7d8159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.17:6789/0 2024-03-29T11:28:33.214 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.221+0000 7f7d8159a640 1 -- 172.21.15.17:0/2664991829 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d840aaf90 con 0x7f7d840b38c0 2024-03-29T11:28:33.214 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.221+0000 7f7d8159a640 10 monclient: _finish_auth 0 2024-03-29T11:28:33.214 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.221+0000 7f7d8159a640 10 monclient: _check_auth_tickets 2024-03-29T11:28:33.214 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.221+0000 7f7d8159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:33.214 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.221+0000 7f7d8159a640 1 -- 172.21.15.17:0/2664991829 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f7d7c001690 con 0x7f7d840b38c0 2024-03-29T11:28:33.215 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.221+0000 7f7d8159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:33.215 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.221+0000 7f7d8159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:33.215 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.221+0000 7f7d8159a640 10 monclient: dump: 2024-03-29T11:28:33.215 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:28:33.215 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:28:33.215 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:33.215 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:33.215 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:33.215 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:33.215 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:28:33.215 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:28:33.215 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:28:33.215 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:33.215 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.221+0000 7f7d8359e640 5 monclient: authenticate success, global_id 4536 2024-03-29T11:28:33.215 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.221+0000 7f7d8359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:33.216 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.221+0000 7f7d8159a640 1 -- 172.21.15.17:0/2664991829 <== mon.0 v1:172.21.15.17:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d7c0017f0 con 0x7f7d840b38c0 2024-03-29T11:28:33.216 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.221+0000 7f7d8159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:33.216 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.221+0000 7f7d8159a640 1 -- 172.21.15.17:0/2664991829 <== mon.0 v1:172.21.15.17:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f7d7c001bc0 con 0x7f7d840b38c0 2024-03-29T11:28:33.216 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.221+0000 7f7d8159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:33.216 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.221+0000 7f7d8159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:33.216 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.221+0000 7f7d8159a640 10 monclient: dump: 2024-03-29T11:28:33.216 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:28:33.216 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:28:33.216 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:33.216 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:33.216 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:33.216 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:33.216 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:28:33.216 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:28:33.216 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:28:33.217 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:33.217 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.221+0000 7f7d8359e640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:33.217 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.221+0000 7f7d8359e640 4 set_mon_vals no callback set 2024-03-29T11:28:33.217 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.222+0000 7f7d8359e640 10 monclient: shutdown 2024-03-29T11:28:33.217 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.222+0000 7f7d8359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:33.217 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.222+0000 7f7d8359e640 1 -- 172.21.15.17:0/2664991829 >> v1:172.21.15.17:6789/0 conn(0x7f7d840b38c0 legacy=0x7f7d8414d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:33.217 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.222+0000 7f7d8359e640 1 -- 172.21.15.17:0/2664991829 shutdown_connections 2024-03-29T11:28:33.217 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.222+0000 7f7d8359e640 1 -- 172.21.15.17:0/2664991829 >> 172.21.15.17:0/2664991829 conn(0x7f7d84066a00 msgr2=0x7f7d84066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:33.217 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.222+0000 7f7d8359e640 1 -- 172.21.15.17:0/2664991829 shutdown_connections 2024-03-29T11:28:33.217 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.222+0000 7f7d8359e640 1 -- 172.21.15.17:0/2664991829 wait complete. 2024-03-29T11:28:33.217 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.223+0000 7f7d8359e640 10 monclient: build_initial_monmap 2024-03-29T11:28:33.217 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.223+0000 7f7d8359e640 10 monclient: monmap: 2024-03-29T11:28:33.217 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:28:33.217 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:33.217 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:28:33.217 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:28:33.218 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:33.218 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:33.218 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:28:33.218 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:28:33.218 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:28:33.218 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:33.220 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.224+0000 7f7d8359e640 1 librados: starting msgr at 2024-03-29T11:28:33.221 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.224+0000 7f7d8359e640 1 librados: starting objecter 2024-03-29T11:28:33.221 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.224+0000 7f7d8359e640 1 Processor -- start 2024-03-29T11:28:33.221 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.224+0000 7f7d8359e640 1 -- start start 2024-03-29T11:28:33.221 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.224+0000 7f7d8359e640 1 librados: setting wanted keys 2024-03-29T11:28:33.221 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.224+0000 7f7d8359e640 1 librados: calling monclient init 2024-03-29T11:28:33.221 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.224+0000 7f7d8359e640 10 monclient: init 2024-03-29T11:28:33.221 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.225+0000 7f7d8359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:33.221 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.225+0000 7f7d8359e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:28:33.221 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.225+0000 7f7d8359e640 10 monclient(hunting): picked mon.noname-a con 0x7f7d840a9c00 addr v1:172.21.15.17:6789/0 2024-03-29T11:28:33.221 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.225+0000 7f7d8359e640 10 monclient(hunting): picked mon.noname-b con 0x7f7d840b2800 addr v1:172.21.15.94:6789/0 2024-03-29T11:28:33.221 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.225+0000 7f7d8359e640 10 monclient(hunting): picked mon.noname-c con 0x7f7d840b38c0 addr v1:172.21.15.17:6790/0 2024-03-29T11:28:33.221 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.225+0000 7f7d8259c640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f7d840a9c00 0x7f7d841de900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:37594/0 (socket says 172.21.15.17:37594) 2024-03-29T11:28:33.221 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.225+0000 7f7d8259c640 1 -- 172.21.15.17:0/2860147431 learned_addr learned my addr 172.21.15.17:0/2860147431 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:28:33.222 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.225+0000 7f7d7affd640 10 client.?.objecter ms_handle_connect 0x7f7d840a9c00 2024-03-29T11:28:33.222 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.225+0000 7f7d7affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:33.222 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.225+0000 7f7d8359e640 1 -- 172.21.15.17:0/2860147431 --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d8413b740 con 0x7f7d840a9c00 2024-03-29T11:28:33.222 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.225+0000 7f7d7affd640 10 client.?.objecter ms_handle_connect 0x7f7d840b38c0 2024-03-29T11:28:33.222 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.225+0000 7f7d7affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:33.222 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.225+0000 7f7d8359e640 1 -- 172.21.15.17:0/2860147431 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d840ab6f0 con 0x7f7d840b2800 2024-03-29T11:28:33.222 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.225+0000 7f7d8359e640 1 -- 172.21.15.17:0/2860147431 --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d84070ef0 con 0x7f7d840b38c0 2024-03-29T11:28:33.222 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.225+0000 7f7d8359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:33.222 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.225+0000 7f7d8359e640 10 monclient(hunting): authenticate will time out at 1419.430685s 2024-03-29T11:28:33.222 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.225+0000 7f7d7affd640 1 -- 172.21.15.17:0/2860147431 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2782099876 0 0) 0x7f7d8413b740 con 0x7f7d840a9c00 2024-03-29T11:28:33.222 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.225+0000 7f7d7affd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:33.222 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.225+0000 7f7d7affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:33.222 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.225+0000 7f7d7affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:33.222 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.226+0000 7f7d7affd640 10 monclient(hunting): my global_id is 4539 2024-03-29T11:28:33.222 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.226+0000 7f7d7affd640 1 -- 172.21.15.17:0/2860147431 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d64004eb0 con 0x7f7d840a9c00 2024-03-29T11:28:33.223 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.226+0000 7f7d7affd640 1 -- 172.21.15.17:0/2860147431 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 272328379 0 0) 0x7f7d84070ef0 con 0x7f7d840b38c0 2024-03-29T11:28:33.223 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.226+0000 7f7d7affd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:33.223 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.226+0000 7f7d7affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:33.223 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.226+0000 7f7d7affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:33.223 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.226+0000 7f7d7affd640 10 monclient(hunting): my global_id is 4628 2024-03-29T11:28:33.223 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.226+0000 7f7d7affd640 1 -- 172.21.15.17:0/2860147431 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d8413b740 con 0x7f7d840b38c0 2024-03-29T11:28:33.223 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.226+0000 7f7d7affd640 10 client.?.objecter ms_handle_connect 0x7f7d840b2800 2024-03-29T11:28:33.223 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.226+0000 7f7d7affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:33.223 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.226+0000 7f7d7affd640 1 -- 172.21.15.17:0/2860147431 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3484729892 0 0) 0x7f7d64004eb0 con 0x7f7d840a9c00 2024-03-29T11:28:33.223 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.226+0000 7f7d7affd640 1 -- 172.21.15.17:0/2860147431 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d84070ef0 con 0x7f7d840a9c00 2024-03-29T11:28:33.223 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.226+0000 7f7d7affd640 1 -- 172.21.15.17:0/2860147431 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f7d6c001960 con 0x7f7d840a9c00 2024-03-29T11:28:33.223 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.226+0000 7f7d7affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:33.223 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.226+0000 7f7d7affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:33.223 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.226+0000 7f7d7affd640 10 monclient(hunting): dump: 2024-03-29T11:28:33.223 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:28:33.224 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:28:33.224 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:33.224 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:33.224 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:33.224 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:33.224 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:28:33.224 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:28:33.224 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:28:33.224 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:33.224 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.226+0000 7f7d7affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:33.224 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.226+0000 7f7d7affd640 1 -- 172.21.15.17:0/2860147431 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1482614776 0 0) 0x7f7d840ab6f0 con 0x7f7d840b2800 2024-03-29T11:28:33.224 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.226+0000 7f7d7affd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:33.224 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.226+0000 7f7d7affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:33.224 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.226+0000 7f7d7affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:33.224 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.226+0000 7f7d7affd640 10 monclient(hunting): my global_id is 4579 2024-03-29T11:28:33.225 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.226+0000 7f7d7affd640 1 -- 172.21.15.17:0/2860147431 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d64004eb0 con 0x7f7d840b2800 2024-03-29T11:28:33.225 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.226+0000 7f7d7affd640 1 -- 172.21.15.17:0/2860147431 <== mon.0 v1:172.21.15.17:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1395445632 0 0) 0x7f7d84070ef0 con 0x7f7d840a9c00 2024-03-29T11:28:33.225 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.226+0000 7f7d7affd640 1 -- 172.21.15.17:0/2860147431 >> v1:172.21.15.17:6790/0 conn(0x7f7d840b38c0 legacy=0x7f7d8413b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:33.225 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.226+0000 7f7d7affd640 1 -- 172.21.15.17:0/2860147431 >> v1:172.21.15.94:6789/0 conn(0x7f7d840b2800 legacy=0x7f7d8413a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:33.225 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.226+0000 7f7d7affd640 10 monclient: _finish_hunting 0 2024-03-29T11:28:33.225 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.226+0000 7f7d7affd640 1 monclient: found mon.a 2024-03-29T11:28:33.225 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.226+0000 7f7d7affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:33.225 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.226+0000 7f7d7affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:28:33.225 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.226+0000 7f7d7affd640 1 -- 172.21.15.17:0/2860147431 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d84072020 con 0x7f7d840a9c00 2024-03-29T11:28:33.225 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.226+0000 7f7d7affd640 10 monclient: _finish_auth 0 2024-03-29T11:28:33.225 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.226+0000 7f7d7affd640 10 monclient: _check_auth_tickets 2024-03-29T11:28:33.225 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.226+0000 7f7d7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:33.225 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.227+0000 7f7d8359e640 5 monclient: authenticate success, global_id 4539 2024-03-29T11:28:33.225 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.227+0000 7f7d8359e640 10 monclient: _renew_subs 2024-03-29T11:28:33.225 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.227+0000 7f7d8359e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:28:33.226 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.227+0000 7f7d8359e640 1 -- 172.21.15.17:0/2860147431 --> v1:172.21.15.17:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d84070260 con 0x7f7d840a9c00 2024-03-29T11:28:33.226 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.227+0000 7f7d8359e640 10 client.4539.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:33.226 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.227+0000 7f7d8359e640 10 monclient: _renew_subs 2024-03-29T11:28:33.226 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.227+0000 7f7d8359e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:28:33.226 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.227+0000 7f7d8359e640 1 -- 172.21.15.17:0/2860147431 --> v1:172.21.15.17:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d841df1f0 con 0x7f7d840a9c00 2024-03-29T11:28:33.226 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.227+0000 7f7d8359e640 1 librados: init done 2024-03-29T11:28:33.226 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.227+0000 7f7d7affd640 1 -- 172.21.15.17:0/2860147431 <== mon.0 v1:172.21.15.17:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d6c0035f0 con 0x7f7d840a9c00 2024-03-29T11:28:33.226 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.227+0000 7f7d7affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:33.226 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.227+0000 7f7d80d99640 4 set_mon_vals no callback set 2024-03-29T11:28:33.226 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.228+0000 7f7d7affd640 1 -- 172.21.15.17:0/2860147431 <== mon.0 v1:172.21.15.17:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f7d6c003940 con 0x7f7d840a9c00 2024-03-29T11:28:33.226 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.228+0000 7f7d7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:33.226 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.228+0000 7f7d7affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:33.226 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.228+0000 7f7d7affd640 10 monclient: dump: 2024-03-29T11:28:33.226 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:28:33.227 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:28:33.227 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:33.227 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:33.227 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:33.227 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:33.227 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:28:33.227 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:28:33.227 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:28:33.227 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:33.227 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.231+0000 7f7d8359e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:33.228 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.231+0000 7f7d8359e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:33.228 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.231+0000 7f7d8359e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:28:33.228 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.232+0000 7f7d8359e640 1 -- 172.21.15.17:0/2860147431 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d84071080 con 0x7f7d840a9c00 2024-03-29T11:28:33.229 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.232+0000 7f7d7affd640 1 -- 172.21.15.17:0/2860147431 <== mon.0 v1:172.21.15.17:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f7d6c003aa0 con 0x7f7d840a9c00 2024-03-29T11:28:33.229 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.233+0000 7f7d7affd640 1 -- 172.21.15.17:0/2860147431 <== mon.0 v1:172.21.15.17:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f7d6c084a50 con 0x7f7d840a9c00 2024-03-29T11:28:33.229 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.233+0000 7f7d7affd640 10 client.4539.objecter ms_dispatch 0x7f7d84000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:33.229 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.233+0000 7f7d7affd640 3 client.4539.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:33.229 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.233+0000 7f7d7affd640 3 client.4539.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:33.229 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.234+0000 7f7d81d9b640 10 monclient: get_auth_request con 0x7f7d6406ddf0 auth_method 0 2024-03-29T11:28:33.229 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.234+0000 7f7d7affd640 20 client.4539.objecter dump_active .. 0 homeless 2024-03-29T11:28:33.229 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.234+0000 7f7d7affd640 10 client.4539.objecter ms_handle_connect 0x7f7d6406ddf0 2024-03-29T11:28:33.230 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.240+0000 7f7d7affd640 1 -- 172.21.15.17:0/2860147431 <== mon.0 v1:172.21.15.17:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7d6c050390 con 0x7f7d840a9c00 2024-03-29T11:28:33.230 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.240+0000 7f7d7affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:33.230 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.240+0000 7f7d7affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:33.597 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.608+0000 7f7d8359e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:33.598 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.608+0000 7f7d8359e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:33.598 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.608+0000 7f7d8359e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:28:33.598 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.608+0000 7f7d8359e640 1 -- 172.21.15.17:0/2860147431 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d840703f0 con 0x7f7d840a9c00 2024-03-29T11:28:33.598 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:28:33.598 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:33.599 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.608+0000 7f7d7affd640 1 -- 172.21.15.17:0/2860147431 <== mon.0 v1:172.21.15.17:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f7d6c084100 con 0x7f7d840a9c00 2024-03-29T11:28:33.599 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.608+0000 7f7d7affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:33.599 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.608+0000 7f7d7affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:33.602 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.611+0000 7f7d8359e640 10 librados: watch_flush enter 2024-03-29T11:28:33.602 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.611+0000 7f7d8359e640 10 librados: watch_flush exit 2024-03-29T11:28:33.602 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.612+0000 7f7d8359e640 20 client.4539.objecter shutdown clearing up homeless session... 2024-03-29T11:28:33.602 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.612+0000 7f7d8359e640 10 client.4539.objecter successfully canceled tick 2024-03-29T11:28:33.602 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.612+0000 7f7d8359e640 1 -- 172.21.15.17:0/2860147431 >> v1:172.21.15.17:6816/3236185052 conn(0x7f7d6406ddf0 legacy=0x7f7d640702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:33.602 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.612+0000 7f7d8359e640 10 monclient: shutdown 2024-03-29T11:28:33.602 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.612+0000 7f7d8359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:33.602 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.612+0000 7f7d8359e640 1 -- 172.21.15.17:0/2860147431 >> v1:172.21.15.17:6789/0 conn(0x7f7d840a9c00 legacy=0x7f7d841de900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:33.602 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.612+0000 7f7d8359e640 1 -- 172.21.15.17:0/2860147431 shutdown_connections 2024-03-29T11:28:33.602 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.612+0000 7f7d8359e640 1 -- 172.21.15.17:0/2860147431 >> 172.21.15.17:0/2860147431 conn(0x7f7d84066a00 msgr2=0x7f7d8413e490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:33.602 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.612+0000 7f7d8359e640 1 -- 172.21.15.17:0/2860147431 shutdown_connections 2024-03-29T11:28:33.603 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.612+0000 7f7d8359e640 1 -- 172.21.15.17:0/2860147431 wait complete. 2024-03-29T11:28:33.603 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:33.612+0000 7f7d8359e640 1 librados: shutdown 2024-03-29T11:28:33.616 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:40.424 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:40.573 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.581+0000 7f670f9f6640 10 monclient: get_monmap_and_config 2024-03-29T11:28:40.573 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.581+0000 7f670f9f6640 10 monclient: build_initial_monmap 2024-03-29T11:28:40.573 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.581+0000 7f670f9f6640 10 monclient: monmap: 2024-03-29T11:28:40.573 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:28:40.573 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:28:40.573 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:28:40.582929+0000 2024-03-29T11:28:40.573 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:28:40.582929+0000 2024-03-29T11:28:40.573 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:40.573 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:40.573 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:28:40.573 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:28:40.573 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:28:40.574 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:40.574 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.584+0000 7f670f9f6640 1 Processor -- start 2024-03-29T11:28:40.574 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.585+0000 7f670f9f6640 1 -- start start 2024-03-29T11:28:40.575 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.585+0000 7f670f9f6640 10 monclient: init 2024-03-29T11:28:40.577 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.585+0000 7f670f9f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:40.577 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.585+0000 7f670f9f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:28:40.577 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.585+0000 7f670f9f6640 10 monclient(hunting): picked mon.noname-c con 0x7f67080ae250 addr v1:172.21.15.94:6789/0 2024-03-29T11:28:40.577 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.585+0000 7f670f9f6640 10 monclient(hunting): picked mon.noname-a con 0x7f670813f970 addr v1:172.21.15.17:6789/0 2024-03-29T11:28:40.577 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.586+0000 7f670f9f6640 10 monclient(hunting): picked mon.noname-b con 0x7f6708142800 addr v1:172.21.15.17:6790/0 2024-03-29T11:28:40.577 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.586+0000 7f670f9f6640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67080afe00 con 0x7f670813f970 2024-03-29T11:28:40.577 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.586+0000 7f670f9f6640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6708142c10 con 0x7f67080ae250 2024-03-29T11:28:40.577 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.586+0000 7f670f9f6640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67080afb20 con 0x7f6708142800 2024-03-29T11:28:40.577 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.586+0000 7f670f9f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:40.577 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.586+0000 7f670f9f6640 10 monclient(hunting): authenticate will time out at 1426.790921s 2024-03-29T11:28:40.581 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.590+0000 7f670d792640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f67080ae250 0x7f670813f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:48601/0 (socket says 172.21.15.17:48601) 2024-03-29T11:28:40.581 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.590+0000 7f670d792640 1 -- 172.21.15.17:0/3984454040 learned_addr learned my addr 172.21.15.17:0/3984454040 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:28:40.581 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.591+0000 7f66fbfff640 1 -- 172.21.15.17:0/3984454040 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2704310051 0 0) 0x7f6708142c10 con 0x7f67080ae250 2024-03-29T11:28:40.581 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.591+0000 7f66fbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:40.581 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.591+0000 7f66fbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:40.581 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.591+0000 7f66fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:40.582 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.591+0000 7f66fbfff640 10 monclient(hunting): my global_id is 4582 2024-03-29T11:28:40.582 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.591+0000 7f66fbfff640 1 -- 172.21.15.17:0/3984454040 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66f4003520 con 0x7f67080ae250 2024-03-29T11:28:40.582 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.590+0000 7f670cf91640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f670813f970 0x7f67081420f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:51039/0 (socket says 172.21.15.17:51039) 2024-03-29T11:28:40.586 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.591+0000 7f66fbfff640 1 -- 172.21.15.17:0/3984454040 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 742978091 0 0) 0x7f67080afe00 con 0x7f670813f970 2024-03-29T11:28:40.586 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.591+0000 7f66fbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:40.586 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.591+0000 7f66fbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:40.586 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.591+0000 7f66fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:40.586 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.591+0000 7f66fbfff640 10 monclient(hunting): my global_id is 4542 2024-03-29T11:28:40.586 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.591+0000 7f66fbfff640 1 -- 172.21.15.17:0/3984454040 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6708142c10 con 0x7f670813f970 2024-03-29T11:28:40.586 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.592+0000 7f66fbfff640 1 -- 172.21.15.17:0/3984454040 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1727355596 0 0) 0x7f6708142c10 con 0x7f670813f970 2024-03-29T11:28:40.586 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.592+0000 7f66fbfff640 1 -- 172.21.15.17:0/3984454040 >> v1:172.21.15.17:6790/0 conn(0x7f6708142800 legacy=0x7f670814da10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:40.586 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.592+0000 7f66fbfff640 1 -- 172.21.15.17:0/3984454040 >> v1:172.21.15.94:6789/0 conn(0x7f67080ae250 legacy=0x7f670813f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:40.586 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.592+0000 7f66fbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:28:40.586 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.592+0000 7f66fbfff640 1 monclient: found mon.noname-a 2024-03-29T11:28:40.586 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.592+0000 7f66fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:40.587 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.592+0000 7f66fbfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.17:6789/0 2024-03-29T11:28:40.587 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.592+0000 7f66fbfff640 1 -- 172.21.15.17:0/3984454040 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67080aab80 con 0x7f670813f970 2024-03-29T11:28:40.587 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.592+0000 7f66fbfff640 10 monclient: _finish_auth 0 2024-03-29T11:28:40.587 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.592+0000 7f66fbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:28:40.587 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.592+0000 7f66fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:40.587 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.592+0000 7f66fbfff640 1 -- 172.21.15.17:0/3984454040 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f67000019f0 con 0x7f670813f970 2024-03-29T11:28:40.587 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.592+0000 7f66fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:40.587 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.592+0000 7f66fbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:40.587 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.592+0000 7f66fbfff640 10 monclient: dump: 2024-03-29T11:28:40.587 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:28:40.587 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:28:40.587 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:40.587 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:40.587 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:40.587 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:40.588 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:28:40.588 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:28:40.588 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:28:40.588 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:40.588 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.592+0000 7f670f9f6640 5 monclient: authenticate success, global_id 4542 2024-03-29T11:28:40.588 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.593+0000 7f670f9f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:40.588 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.593+0000 7f66fbfff640 1 -- 172.21.15.17:0/3984454040 <== mon.0 v1:172.21.15.17:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6700001b50 con 0x7f670813f970 2024-03-29T11:28:40.588 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.593+0000 7f66fbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:40.588 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.593+0000 7f66fbfff640 1 -- 172.21.15.17:0/3984454040 <== mon.0 v1:172.21.15.17:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f6700001f60 con 0x7f670813f970 2024-03-29T11:28:40.588 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.593+0000 7f66fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:40.588 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.593+0000 7f66fbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:40.588 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.593+0000 7f66fbfff640 10 monclient: dump: 2024-03-29T11:28:40.588 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:28:40.588 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:28:40.588 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:40.589 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:40.589 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:40.589 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:40.589 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:28:40.589 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:28:40.589 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:28:40.589 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:40.589 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.593+0000 7f670f9f6640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:40.589 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.593+0000 7f670f9f6640 4 set_mon_vals no callback set 2024-03-29T11:28:40.589 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.594+0000 7f670f9f6640 10 monclient: shutdown 2024-03-29T11:28:40.589 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.594+0000 7f670f9f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:40.589 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.594+0000 7f670f9f6640 1 -- 172.21.15.17:0/3984454040 >> v1:172.21.15.17:6789/0 conn(0x7f670813f970 legacy=0x7f67081420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:40.589 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.594+0000 7f670f9f6640 1 -- 172.21.15.17:0/3984454040 shutdown_connections 2024-03-29T11:28:40.589 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.594+0000 7f670f9f6640 1 -- 172.21.15.17:0/3984454040 >> 172.21.15.17:0/3984454040 conn(0x7f67080b35e0 msgr2=0x7f67080b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:40.589 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.596+0000 7f670f9f6640 1 -- 172.21.15.17:0/3984454040 shutdown_connections 2024-03-29T11:28:40.590 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.597+0000 7f670f9f6640 1 -- 172.21.15.17:0/3984454040 wait complete. 2024-03-29T11:28:40.591 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.597+0000 7f670f9f6640 10 monclient: build_initial_monmap 2024-03-29T11:28:40.591 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.597+0000 7f670f9f6640 10 monclient: monmap: 2024-03-29T11:28:40.591 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:28:40.591 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:40.591 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:28:40.591 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:28:40.591 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:40.591 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:40.591 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:28:40.591 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:28:40.591 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:28:40.591 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:40.591 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.598+0000 7f670f9f6640 1 librados: starting msgr at 2024-03-29T11:28:40.591 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.598+0000 7f670f9f6640 1 librados: starting objecter 2024-03-29T11:28:40.592 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.598+0000 7f670f9f6640 1 Processor -- start 2024-03-29T11:28:40.592 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.598+0000 7f670f9f6640 1 -- start start 2024-03-29T11:28:40.594 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.598+0000 7f670f9f6640 1 librados: setting wanted keys 2024-03-29T11:28:40.594 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.598+0000 7f670f9f6640 1 librados: calling monclient init 2024-03-29T11:28:40.594 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.598+0000 7f670f9f6640 10 monclient: init 2024-03-29T11:28:40.594 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.599+0000 7f670f9f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:40.594 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.599+0000 7f670f9f6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:28:40.595 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.600+0000 7f670f9f6640 10 monclient(hunting): picked mon.noname-c con 0x7f67080ae250 addr v1:172.21.15.17:6790/0 2024-03-29T11:28:40.595 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.603+0000 7f670f9f6640 10 monclient(hunting): picked mon.noname-a con 0x7f6708142800 addr v1:172.21.15.17:6789/0 2024-03-29T11:28:40.595 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.603+0000 7f670f9f6640 10 monclient(hunting): picked mon.noname-b con 0x7f67081df740 addr v1:172.21.15.94:6789/0 2024-03-29T11:28:40.595 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.603+0000 7f670f9f6640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67081dfbb0 con 0x7f6708142800 2024-03-29T11:28:40.595 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.603+0000 7f670f9f6640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67081453f0 con 0x7f67081df740 2024-03-29T11:28:40.595 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.603+0000 7f670f9f6640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6708143df0 con 0x7f67080ae250 2024-03-29T11:28:40.595 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.603+0000 7f670f9f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:40.595 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.604+0000 7f670f9f6640 10 monclient(hunting): authenticate will time out at 1426.808883s 2024-03-29T11:28:40.595 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.604+0000 7f670cf91640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f6708142800 0x7f67081df030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:51415/0 (socket says 172.21.15.17:51415) 2024-03-29T11:28:40.595 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.604+0000 7f670cf91640 1 -- 172.21.15.17:0/808021985 learned_addr learned my addr 172.21.15.17:0/808021985 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:28:40.596 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.605+0000 7f66f9ffb640 10 client.?.objecter ms_handle_connect 0x7f6708142800 2024-03-29T11:28:40.597 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.605+0000 7f66f9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:40.597 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.605+0000 7f66f9ffb640 1 -- 172.21.15.17:0/808021985 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3667141015 0 0) 0x7f67081dfbb0 con 0x7f6708142800 2024-03-29T11:28:40.597 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.605+0000 7f66f9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:40.597 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.605+0000 7f66f9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:40.597 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.605+0000 7f66f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:40.597 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.605+0000 7f66f9ffb640 10 monclient(hunting): my global_id is 4545 2024-03-29T11:28:40.597 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.605+0000 7f66f9ffb640 1 -- 172.21.15.17:0/808021985 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66ec004eb0 con 0x7f6708142800 2024-03-29T11:28:40.597 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.605+0000 7f66f9ffb640 10 client.?.objecter ms_handle_connect 0x7f67080ae250 2024-03-29T11:28:40.597 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.605+0000 7f66f9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:40.597 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.606+0000 7f66f9ffb640 1 -- 172.21.15.17:0/808021985 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1535538765 0 0) 0x7f6708143df0 con 0x7f67080ae250 2024-03-29T11:28:40.597 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.606+0000 7f66f9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:40.597 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.606+0000 7f66f9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:40.597 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.606+0000 7f66f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:40.598 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.606+0000 7f66f9ffb640 10 monclient(hunting): my global_id is 4631 2024-03-29T11:28:40.598 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.606+0000 7f66f9ffb640 1 -- 172.21.15.17:0/808021985 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67081dfbb0 con 0x7f67080ae250 2024-03-29T11:28:40.598 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.606+0000 7f66f9ffb640 1 -- 172.21.15.17:0/808021985 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3583536988 0 0) 0x7f66ec004eb0 con 0x7f6708142800 2024-03-29T11:28:40.598 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.606+0000 7f66f9ffb640 1 -- 172.21.15.17:0/808021985 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6708143df0 con 0x7f6708142800 2024-03-29T11:28:40.598 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.606+0000 7f66f9ffb640 1 -- 172.21.15.17:0/808021985 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f6700001f60 con 0x7f6708142800 2024-03-29T11:28:40.598 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.606+0000 7f66f9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:40.598 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.606+0000 7f66f9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:40.598 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.606+0000 7f66f9ffb640 10 monclient(hunting): dump: 2024-03-29T11:28:40.598 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:28:40.598 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:28:40.598 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:40.598 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:40.598 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:40.599 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:40.599 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:28:40.599 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:28:40.599 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:28:40.599 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:40.599 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.606+0000 7f66f9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:40.599 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.606+0000 7f66f9ffb640 1 -- 172.21.15.17:0/808021985 <== mon.0 v1:172.21.15.17:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2914009218 0 0) 0x7f6708143df0 con 0x7f6708142800 2024-03-29T11:28:40.599 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.606+0000 7f66f9ffb640 1 -- 172.21.15.17:0/808021985 >> v1:172.21.15.17:6790/0 conn(0x7f67080ae250 legacy=0x7f67081436e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:40.599 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.606+0000 7f66f9ffb640 1 -- 172.21.15.17:0/808021985 >> v1:172.21.15.94:6789/0 conn(0x7f67081df740 legacy=0x7f67081e65d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:40.601 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.609+0000 7f66f9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:28:40.601 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.609+0000 7f66f9ffb640 1 monclient: found mon.a 2024-03-29T11:28:40.601 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.609+0000 7f66f9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:40.601 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.609+0000 7f66f9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:28:40.601 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.609+0000 7f66f9ffb640 1 -- 172.21.15.17:0/808021985 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6708145660 con 0x7f6708142800 2024-03-29T11:28:40.601 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.609+0000 7f66f9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:28:40.601 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.609+0000 7f66f9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:28:40.601 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.609+0000 7f66f9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:40.602 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.609+0000 7f670f9f6640 5 monclient: authenticate success, global_id 4545 2024-03-29T11:28:40.602 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.609+0000 7f670f9f6640 10 monclient: _renew_subs 2024-03-29T11:28:40.602 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.609+0000 7f670f9f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:28:40.602 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.610+0000 7f670f9f6640 1 -- 172.21.15.17:0/808021985 --> v1:172.21.15.17:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6708145f00 con 0x7f6708142800 2024-03-29T11:28:40.602 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.610+0000 7f670f9f6640 10 client.4545.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:40.602 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.610+0000 7f670f9f6640 10 monclient: _renew_subs 2024-03-29T11:28:40.602 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.610+0000 7f670f9f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:28:40.602 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.610+0000 7f670f9f6640 1 -- 172.21.15.17:0/808021985 --> v1:172.21.15.17:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6708144770 con 0x7f6708142800 2024-03-29T11:28:40.602 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.610+0000 7f670f9f6640 1 librados: init done 2024-03-29T11:28:40.603 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.611+0000 7f66f9ffb640 10 client.4545.objecter ms_handle_connect 0x7f67081df740 2024-03-29T11:28:40.603 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.611+0000 7f66f9ffb640 10 client.4545.objecter resend_mon_ops 2024-03-29T11:28:40.603 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.611+0000 7f670f9f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:40.603 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.611+0000 7f670f9f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:40.603 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.612+0000 7f670f9f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:28:40.603 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.612+0000 7f670f9f6640 1 -- 172.21.15.17:0/808021985 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6708146140 con 0x7f6708142800 2024-03-29T11:28:40.605 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.615+0000 7f66f9ffb640 1 -- 172.21.15.17:0/808021985 <== mon.0 v1:172.21.15.17:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6700004e10 con 0x7f6708142800 2024-03-29T11:28:40.605 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.615+0000 7f66f9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:40.605 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.615+0000 7f66f9ffb640 1 -- 172.21.15.17:0/808021985 <== mon.0 v1:172.21.15.17:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f6700005040 con 0x7f6708142800 2024-03-29T11:28:40.605 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.615+0000 7f66f9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:40.605 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.615+0000 7f66f9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:40.605 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.615+0000 7f66f9ffb640 10 monclient: dump: 2024-03-29T11:28:40.605 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:28:40.606 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:28:40.606 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:40.606 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:40.606 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:40.606 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:40.606 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:28:40.606 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:28:40.606 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:28:40.606 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:40.606 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.615+0000 7f66fb7fe640 4 set_mon_vals no callback set 2024-03-29T11:28:40.607 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.616+0000 7f66f9ffb640 1 -- 172.21.15.17:0/808021985 <== mon.0 v1:172.21.15.17:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f6700005260 con 0x7f6708142800 2024-03-29T11:28:40.610 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.620+0000 7f66f9ffb640 1 -- 172.21.15.17:0/808021985 <== mon.0 v1:172.21.15.17:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7f6700084d50 con 0x7f6708142800 2024-03-29T11:28:40.610 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.620+0000 7f66f9ffb640 10 client.4545.objecter ms_dispatch 0x7f6708000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:40.610 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.620+0000 7f66f9ffb640 3 client.4545.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:40.611 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.620+0000 7f66f9ffb640 3 client.4545.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:40.611 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.620+0000 7f66f9ffb640 20 client.4545.objecter dump_active .. 0 homeless 2024-03-29T11:28:40.613 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.623+0000 7f670d792640 10 monclient: get_auth_request con 0x7f66ec06dd10 auth_method 0 2024-03-29T11:28:40.613 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.623+0000 7f66f9ffb640 1 -- 172.21.15.17:0/808021985 <== mon.0 v1:172.21.15.17:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f67000507b0 con 0x7f6708142800 2024-03-29T11:28:40.614 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.623+0000 7f66f9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:40.614 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.623+0000 7f66f9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:40.615 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.625+0000 7f66f9ffb640 10 client.4545.objecter ms_handle_connect 0x7f66ec06dd10 2024-03-29T11:28:40.945 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.954+0000 7f670f9f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:40.945 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.954+0000 7f670f9f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:40.945 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.954+0000 7f670f9f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:28:40.945 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.954+0000 7f670f9f6640 1 -- 172.21.15.17:0/808021985 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6708075f10 con 0x7f6708142800 2024-03-29T11:28:40.946 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.956+0000 7f66f9ffb640 1 -- 172.21.15.17:0/808021985 <== mon.0 v1:172.21.15.17:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7f6700054b60 con 0x7f6708142800 2024-03-29T11:28:40.946 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.956+0000 7f66f9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:40.946 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.956+0000 7f66f9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:40.949 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:28:40.949 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:40.951 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.961+0000 7f670f9f6640 10 librados: watch_flush enter 2024-03-29T11:28:40.951 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.961+0000 7f670f9f6640 10 librados: watch_flush exit 2024-03-29T11:28:40.951 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.961+0000 7f670f9f6640 20 client.4545.objecter shutdown clearing up homeless session... 2024-03-29T11:28:40.952 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.961+0000 7f670f9f6640 10 client.4545.objecter successfully canceled tick 2024-03-29T11:28:40.952 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.962+0000 7f670f9f6640 1 -- 172.21.15.17:0/808021985 >> v1:172.21.15.17:6816/3236185052 conn(0x7f66ec06dd10 legacy=0x7f66ec0701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:40.952 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.962+0000 7f670f9f6640 10 monclient: shutdown 2024-03-29T11:28:40.952 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.962+0000 7f670f9f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:40.953 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.962+0000 7f670f9f6640 1 -- 172.21.15.17:0/808021985 >> v1:172.21.15.17:6789/0 conn(0x7f6708142800 legacy=0x7f67081df030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:40.953 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.962+0000 7f670f9f6640 1 -- 172.21.15.17:0/808021985 shutdown_connections 2024-03-29T11:28:40.953 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.962+0000 7f670f9f6640 1 -- 172.21.15.17:0/808021985 >> 172.21.15.17:0/808021985 conn(0x7f67080b35e0 msgr2=0x7f67080a96f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:40.956 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.966+0000 7f670f9f6640 1 -- 172.21.15.17:0/808021985 shutdown_connections 2024-03-29T11:28:40.956 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.966+0000 7f670f9f6640 1 -- 172.21.15.17:0/808021985 wait complete. 2024-03-29T11:28:40.956 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:40.966+0000 7f670f9f6640 1 librados: shutdown 2024-03-29T11:28:40.974 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:47.781 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:47.929 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.938+0000 7fbc7dde8640 10 monclient: get_monmap_and_config 2024-03-29T11:28:47.929 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.938+0000 7fbc7dde8640 10 monclient: build_initial_monmap 2024-03-29T11:28:47.929 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.938+0000 7fbc7dde8640 10 monclient: monmap: 2024-03-29T11:28:47.930 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:28:47.930 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:28:47.930 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:28:47.939547+0000 2024-03-29T11:28:47.930 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:28:47.939547+0000 2024-03-29T11:28:47.930 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:47.930 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:47.930 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:28:47.930 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:28:47.930 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:28:47.930 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:47.931 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.939+0000 7fbc7dde8640 1 Processor -- start 2024-03-29T11:28:47.931 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.939+0000 7fbc7dde8640 1 -- start start 2024-03-29T11:28:47.931 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.939+0000 7fbc7dde8640 10 monclient: init 2024-03-29T11:28:47.931 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.940+0000 7fbc7dde8640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:47.931 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.940+0000 7fbc7dde8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:28:47.931 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.940+0000 7fbc7dde8640 10 monclient(hunting): picked mon.noname-a con 0x7fbc70034850 addr v1:172.21.15.17:6789/0 2024-03-29T11:28:47.931 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.940+0000 7fbc7dde8640 10 monclient(hunting): picked mon.noname-b con 0x7fbc700c9550 addr v1:172.21.15.17:6790/0 2024-03-29T11:28:47.931 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.940+0000 7fbc7dde8640 10 monclient(hunting): picked mon.noname-c con 0x7fbc7002aac0 addr v1:172.21.15.94:6789/0 2024-03-29T11:28:47.931 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.940+0000 7fbc7dde8640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc7002c440 con 0x7fbc70034850 2024-03-29T11:28:47.931 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.940+0000 7fbc7dde8640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc7002aed0 con 0x7fbc7002aac0 2024-03-29T11:28:47.931 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.940+0000 7fbc7dde8640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc7002b190 con 0x7fbc700c9550 2024-03-29T11:28:47.932 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.940+0000 7fbc7dde8640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:47.932 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.940+0000 7fbc7dde8640 10 monclient(hunting): authenticate will time out at 1434.145687s 2024-03-29T11:28:47.932 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.941+0000 7fbc7cde6640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fbc70034850 0x7fbc700c8e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:58796/0 (socket says 172.21.15.17:58796) 2024-03-29T11:28:47.932 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.941+0000 7fbc7cde6640 1 -- 172.21.15.17:0/1192319168 learned_addr learned my addr 172.21.15.17:0/1192319168 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:28:47.932 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.942+0000 7fbc777fe640 1 -- 172.21.15.17:0/1192319168 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 462144144 0 0) 0x7fbc7002c440 con 0x7fbc70034850 2024-03-29T11:28:47.932 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.942+0000 7fbc777fe640 10 monclient(hunting): _negotiate 2024-03-29T11:28:47.932 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.942+0000 7fbc777fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:47.932 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.942+0000 7fbc777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:47.932 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.942+0000 7fbc777fe640 10 monclient(hunting): my global_id is 4548 2024-03-29T11:28:47.933 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.942+0000 7fbc777fe640 1 -- 172.21.15.17:0/1192319168 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc68003520 con 0x7fbc70034850 2024-03-29T11:28:47.933 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.943+0000 7fbc777fe640 1 -- 172.21.15.17:0/1192319168 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2215448009 0 0) 0x7fbc68003520 con 0x7fbc70034850 2024-03-29T11:28:47.933 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.943+0000 7fbc777fe640 1 -- 172.21.15.17:0/1192319168 >> v1:172.21.15.17:6790/0 conn(0x7fbc700c9550 legacy=0x7fbc700cbea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:47.935 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.944+0000 7fbc777fe640 1 -- 172.21.15.17:0/1192319168 >> v1:172.21.15.94:6789/0 conn(0x7fbc7002aac0 legacy=0x7fbc700cc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:47.935 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.944+0000 7fbc777fe640 10 monclient: _finish_hunting 0 2024-03-29T11:28:47.935 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.944+0000 7fbc777fe640 1 monclient: found mon.noname-a 2024-03-29T11:28:47.935 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.944+0000 7fbc777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:47.935 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.944+0000 7fbc777fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.17:6789/0 2024-03-29T11:28:47.935 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.944+0000 7fbc777fe640 1 -- 172.21.15.17:0/1192319168 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc70033060 con 0x7fbc70034850 2024-03-29T11:28:47.935 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.944+0000 7fbc777fe640 10 monclient: _finish_auth 0 2024-03-29T11:28:47.936 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.944+0000 7fbc777fe640 10 monclient: _check_auth_tickets 2024-03-29T11:28:47.936 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.944+0000 7fbc777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:47.936 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.944+0000 7fbc777fe640 1 -- 172.21.15.17:0/1192319168 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fbc6c001690 con 0x7fbc70034850 2024-03-29T11:28:47.936 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.944+0000 7fbc777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:47.936 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.944+0000 7fbc777fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:47.936 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.944+0000 7fbc777fe640 10 monclient: dump: 2024-03-29T11:28:47.936 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:28:47.936 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:28:47.936 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:47.936 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:47.936 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:47.937 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:47.937 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:28:47.937 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:28:47.937 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:28:47.937 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:47.937 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.944+0000 7fbc7dde8640 5 monclient: authenticate success, global_id 4548 2024-03-29T11:28:47.937 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.944+0000 7fbc7dde8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:47.937 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.944+0000 7fbc777fe640 1 -- 172.21.15.17:0/1192319168 <== mon.0 v1:172.21.15.17:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc6c001860 con 0x7fbc70034850 2024-03-29T11:28:47.937 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.944+0000 7fbc777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:47.937 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.944+0000 7fbc777fe640 1 -- 172.21.15.17:0/1192319168 <== mon.0 v1:172.21.15.17:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fbc6c001cb0 con 0x7fbc70034850 2024-03-29T11:28:47.937 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.944+0000 7fbc777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:47.937 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.944+0000 7fbc777fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:47.937 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.944+0000 7fbc777fe640 10 monclient: dump: 2024-03-29T11:28:47.937 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:28:47.938 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:28:47.938 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:47.938 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:47.938 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:47.938 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:47.938 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:28:47.938 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:28:47.938 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:28:47.938 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:47.938 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.944+0000 7fbc7dde8640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:47.938 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.944+0000 7fbc7dde8640 4 set_mon_vals no callback set 2024-03-29T11:28:47.939 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.945+0000 7fbc7dde8640 10 monclient: shutdown 2024-03-29T11:28:47.939 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.945+0000 7fbc7dde8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:47.939 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.945+0000 7fbc7dde8640 1 -- 172.21.15.17:0/1192319168 >> v1:172.21.15.17:6789/0 conn(0x7fbc70034850 legacy=0x7fbc700c8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:47.940 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.946+0000 7fbc7dde8640 1 -- 172.21.15.17:0/1192319168 shutdown_connections 2024-03-29T11:28:47.940 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.946+0000 7fbc7dde8640 1 -- 172.21.15.17:0/1192319168 >> 172.21.15.17:0/1192319168 conn(0x7fbc700058f0 msgr2=0x7fbc70005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:47.940 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.946+0000 7fbc7dde8640 1 -- 172.21.15.17:0/1192319168 shutdown_connections 2024-03-29T11:28:47.940 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.947+0000 7fbc7dde8640 1 -- 172.21.15.17:0/1192319168 wait complete. 2024-03-29T11:28:47.940 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.947+0000 7fbc7dde8640 10 monclient: build_initial_monmap 2024-03-29T11:28:47.940 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.947+0000 7fbc7dde8640 10 monclient: monmap: 2024-03-29T11:28:47.940 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:28:47.940 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:47.940 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:28:47.940 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:28:47.940 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:47.940 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:47.940 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:28:47.940 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:28:47.941 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:28:47.941 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:47.941 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.948+0000 7fbc7dde8640 1 librados: starting msgr at 2024-03-29T11:28:47.941 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.948+0000 7fbc7dde8640 1 librados: starting objecter 2024-03-29T11:28:47.941 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.949+0000 7fbc7dde8640 1 Processor -- start 2024-03-29T11:28:47.948 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.949+0000 7fbc7dde8640 1 -- start start 2024-03-29T11:28:47.948 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.949+0000 7fbc7dde8640 1 librados: setting wanted keys 2024-03-29T11:28:47.948 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.949+0000 7fbc7dde8640 1 librados: calling monclient init 2024-03-29T11:28:47.948 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.949+0000 7fbc7dde8640 10 monclient: init 2024-03-29T11:28:47.948 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.950+0000 7fbc7dde8640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:47.948 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.950+0000 7fbc7dde8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:28:47.948 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.950+0000 7fbc7dde8640 10 monclient(hunting): picked mon.noname-c con 0x7fbc7002aac0 addr v1:172.21.15.17:6790/0 2024-03-29T11:28:47.948 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.950+0000 7fbc7dde8640 10 monclient(hunting): picked mon.noname-a con 0x7fbc70034850 addr v1:172.21.15.17:6789/0 2024-03-29T11:28:47.948 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.950+0000 7fbc7dde8640 10 monclient(hunting): picked mon.noname-b con 0x7fbc700c9550 addr v1:172.21.15.94:6789/0 2024-03-29T11:28:47.948 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.950+0000 7fbc7dde8640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc700c1150 con 0x7fbc70034850 2024-03-29T11:28:47.949 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.950+0000 7fbc7dde8640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc700c1430 con 0x7fbc700c9550 2024-03-29T11:28:47.949 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.950+0000 7fbc7dde8640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc700c1710 con 0x7fbc7002aac0 2024-03-29T11:28:47.949 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.950+0000 7fbc7dde8640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:47.949 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.950+0000 7fbc7dde8640 10 monclient(hunting): authenticate will time out at 1434.155246s 2024-03-29T11:28:47.949 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.951+0000 7fbc77fff640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fbc70034850 0x7fbc7016bda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:60682/0 (socket says 172.21.15.17:60682) 2024-03-29T11:28:47.949 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.951+0000 7fbc77fff640 1 -- 172.21.15.17:0/3258762294 learned_addr learned my addr 172.21.15.17:0/3258762294 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:28:47.949 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.951+0000 7fbc757fa640 10 client.?.objecter ms_handle_connect 0x7fbc70034850 2024-03-29T11:28:47.949 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.951+0000 7fbc757fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:47.949 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.952+0000 7fbc757fa640 1 -- 172.21.15.17:0/3258762294 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1293425164 0 0) 0x7fbc700c1150 con 0x7fbc70034850 2024-03-29T11:28:47.949 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.952+0000 7fbc757fa640 10 monclient(hunting): _negotiate 2024-03-29T11:28:47.950 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.952+0000 7fbc757fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:47.950 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.952+0000 7fbc757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:47.950 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.952+0000 7fbc757fa640 10 monclient(hunting): my global_id is 4551 2024-03-29T11:28:47.950 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.952+0000 7fbc757fa640 1 -- 172.21.15.17:0/3258762294 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc5c004eb0 con 0x7fbc70034850 2024-03-29T11:28:47.950 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.954+0000 7fbc757fa640 10 client.?.objecter ms_handle_connect 0x7fbc7002aac0 2024-03-29T11:28:47.950 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.954+0000 7fbc757fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:47.950 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.954+0000 7fbc757fa640 1 -- 172.21.15.17:0/3258762294 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3200826495 0 0) 0x7fbc700c1710 con 0x7fbc7002aac0 2024-03-29T11:28:47.950 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.954+0000 7fbc757fa640 10 monclient(hunting): _negotiate 2024-03-29T11:28:47.950 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.954+0000 7fbc757fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:47.950 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.954+0000 7fbc757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:47.951 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.954+0000 7fbc757fa640 10 monclient(hunting): my global_id is 4634 2024-03-29T11:28:47.951 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.954+0000 7fbc757fa640 1 -- 172.21.15.17:0/3258762294 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc700c1150 con 0x7fbc7002aac0 2024-03-29T11:28:47.951 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.954+0000 7fbc757fa640 1 -- 172.21.15.17:0/3258762294 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3403388305 0 0) 0x7fbc5c004eb0 con 0x7fbc70034850 2024-03-29T11:28:47.951 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.954+0000 7fbc757fa640 1 -- 172.21.15.17:0/3258762294 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc700c1710 con 0x7fbc70034850 2024-03-29T11:28:47.951 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.954+0000 7fbc757fa640 1 -- 172.21.15.17:0/3258762294 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fbc640017d0 con 0x7fbc70034850 2024-03-29T11:28:47.951 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.954+0000 7fbc757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:47.951 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.954+0000 7fbc757fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:47.951 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.954+0000 7fbc757fa640 10 monclient(hunting): dump: 2024-03-29T11:28:47.951 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:28:47.951 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:28:47.951 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:47.951 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:47.951 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:47.951 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:47.952 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:28:47.952 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:28:47.952 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:28:47.952 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:47.952 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.954+0000 7fbc757fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:47.952 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.955+0000 7fbc757fa640 1 -- 172.21.15.17:0/3258762294 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2399862428 0 0) 0x7fbc700c1150 con 0x7fbc7002aac0 2024-03-29T11:28:47.952 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.955+0000 7fbc757fa640 1 -- 172.21.15.17:0/3258762294 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc5c004eb0 con 0x7fbc7002aac0 2024-03-29T11:28:47.952 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.955+0000 7fbc757fa640 1 -- 172.21.15.17:0/3258762294 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fbc6c001b70 con 0x7fbc7002aac0 2024-03-29T11:28:47.952 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.955+0000 7fbc757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:47.952 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.955+0000 7fbc757fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:47.952 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.955+0000 7fbc757fa640 10 monclient(hunting): dump: 2024-03-29T11:28:47.952 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:28:47.952 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:28:47.952 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:47.952 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:47.953 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:47.953 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:47.953 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:28:47.953 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:28:47.953 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:28:47.953 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:47.953 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.955+0000 7fbc757fa640 1 -- 172.21.15.17:0/3258762294 <== mon.2 v1:172.21.15.17:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 900421121 0 0) 0x7fbc5c004eb0 con 0x7fbc7002aac0 2024-03-29T11:28:47.953 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.955+0000 7fbc757fa640 1 -- 172.21.15.17:0/3258762294 >> v1:172.21.15.94:6789/0 conn(0x7fbc700c9550 legacy=0x7fbc700bce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:47.953 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.955+0000 7fbc757fa640 1 -- 172.21.15.17:0/3258762294 >> v1:172.21.15.17:6789/0 conn(0x7fbc70034850 legacy=0x7fbc7016bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:47.953 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.955+0000 7fbc757fa640 10 monclient: _finish_hunting 0 2024-03-29T11:28:47.953 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.955+0000 7fbc757fa640 1 monclient: found mon.c 2024-03-29T11:28:47.953 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.955+0000 7fbc757fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:47.953 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.955+0000 7fbc757fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:28:47.953 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.955+0000 7fbc757fa640 1 -- 172.21.15.17:0/3258762294 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc700bede0 con 0x7fbc7002aac0 2024-03-29T11:28:47.954 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.955+0000 7fbc757fa640 10 monclient: _finish_auth 0 2024-03-29T11:28:47.954 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.955+0000 7fbc757fa640 10 monclient: _check_auth_tickets 2024-03-29T11:28:47.954 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.955+0000 7fbc757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:47.954 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.955+0000 7fbc7dde8640 5 monclient: authenticate success, global_id 4634 2024-03-29T11:28:47.954 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.955+0000 7fbc7dde8640 10 monclient: _renew_subs 2024-03-29T11:28:47.954 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.955+0000 7fbc7dde8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:28:47.954 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.955+0000 7fbc7dde8640 1 -- 172.21.15.17:0/3258762294 --> v1:172.21.15.17:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc7016c4b0 con 0x7fbc7002aac0 2024-03-29T11:28:47.954 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.955+0000 7fbc7dde8640 10 client.4634.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:47.954 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.955+0000 7fbc7dde8640 10 monclient: _renew_subs 2024-03-29T11:28:47.954 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.955+0000 7fbc7dde8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:28:47.954 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.955+0000 7fbc7dde8640 1 -- 172.21.15.17:0/3258762294 --> v1:172.21.15.17:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc7016c7b0 con 0x7fbc7002aac0 2024-03-29T11:28:47.954 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.955+0000 7fbc7dde8640 1 librados: init done 2024-03-29T11:28:47.954 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.955+0000 7fbc757fa640 1 -- 172.21.15.17:0/3258762294 <== mon.2 v1:172.21.15.17:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc6c001d30 con 0x7fbc7002aac0 2024-03-29T11:28:47.954 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.955+0000 7fbc757fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:47.955 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.955+0000 7fbc757fa640 1 -- 172.21.15.17:0/3258762294 <== mon.2 v1:172.21.15.17:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fbc6c004d90 con 0x7fbc7002aac0 2024-03-29T11:28:47.955 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.956+0000 7fbc757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:47.955 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.956+0000 7fbc757fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:47.955 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.956+0000 7fbc757fa640 10 monclient: dump: 2024-03-29T11:28:47.955 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:28:47.955 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:28:47.955 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:47.955 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:47.955 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:47.955 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:47.955 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:28:47.955 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:28:47.955 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:28:47.955 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:47.955 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.957+0000 7fbc7dde8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:47.956 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.957+0000 7fbc7dde8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:47.956 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.957+0000 7fbc7dde8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:28:47.956 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.957+0000 7fbc7dde8640 1 -- 172.21.15.17:0/3258762294 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc70004cd0 con 0x7fbc7002aac0 2024-03-29T11:28:47.956 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.957+0000 7fbc767fc640 4 set_mon_vals no callback set 2024-03-29T11:28:47.956 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.958+0000 7fbc757fa640 1 -- 172.21.15.17:0/3258762294 <== mon.2 v1:172.21.15.17:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7fbc6c01ad60 con 0x7fbc7002aac0 2024-03-29T11:28:47.957 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.958+0000 7fbc757fa640 1 -- 172.21.15.17:0/3258762294 <== mon.2 v1:172.21.15.17:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7fbc6c084b00 con 0x7fbc7002aac0 2024-03-29T11:28:47.957 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.958+0000 7fbc757fa640 10 client.4634.objecter ms_dispatch 0x7fbc78000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:47.957 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.958+0000 7fbc757fa640 3 client.4634.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:47.957 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.958+0000 7fbc757fa640 3 client.4634.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:47.957 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.959+0000 7fbc757fa640 20 client.4634.objecter dump_active .. 0 homeless 2024-03-29T11:28:47.957 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.959+0000 7fbc77fff640 10 monclient: get_auth_request con 0x7fbc5c06df70 auth_method 0 2024-03-29T11:28:47.958 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.959+0000 7fbc757fa640 10 client.4634.objecter ms_handle_connect 0x7fbc5c06df70 2024-03-29T11:28:47.958 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.967+0000 7fbc757fa640 1 -- 172.21.15.17:0/3258762294 <== mon.2 v1:172.21.15.17:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbc6c0503e0 con 0x7fbc7002aac0 2024-03-29T11:28:47.958 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.967+0000 7fbc757fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:47.958 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:47.967+0000 7fbc757fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:48.283 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:48.293+0000 7fbc7dde8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:48.284 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:48.293+0000 7fbc7dde8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:48.284 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:48.293+0000 7fbc7dde8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:28:48.284 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:48.293+0000 7fbc7dde8640 1 -- 172.21.15.17:0/3258762294 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbc700c7570 con 0x7fbc7002aac0 2024-03-29T11:28:48.284 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:48.294+0000 7fbc757fa640 1 -- 172.21.15.17:0/3258762294 <== mon.2 v1:172.21.15.17:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7fbc6c052310 con 0x7fbc7002aac0 2024-03-29T11:28:48.284 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:48.294+0000 7fbc757fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:48.284 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:48.294+0000 7fbc757fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:48.288 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:28:48.288 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:48.291 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:48.300+0000 7fbc7dde8640 10 librados: watch_flush enter 2024-03-29T11:28:48.291 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:48.300+0000 7fbc7dde8640 10 librados: watch_flush exit 2024-03-29T11:28:48.291 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:48.300+0000 7fbc7dde8640 20 client.4634.objecter shutdown clearing up homeless session... 2024-03-29T11:28:48.291 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:48.300+0000 7fbc7dde8640 10 client.4634.objecter successfully canceled tick 2024-03-29T11:28:48.291 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:48.300+0000 7fbc7dde8640 1 -- 172.21.15.17:0/3258762294 >> v1:172.21.15.17:6816/3236185052 conn(0x7fbc5c06df70 legacy=0x7fbc5c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:48.291 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:48.300+0000 7fbc7dde8640 10 monclient: shutdown 2024-03-29T11:28:48.291 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:48.300+0000 7fbc7dde8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:48.291 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:48.300+0000 7fbc7dde8640 1 -- 172.21.15.17:0/3258762294 >> v1:172.21.15.17:6790/0 conn(0x7fbc7002aac0 legacy=0x7fbc700bf680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:48.291 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:48.300+0000 7fbc7dde8640 1 -- 172.21.15.17:0/3258762294 shutdown_connections 2024-03-29T11:28:48.291 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:48.300+0000 7fbc7dde8640 1 -- 172.21.15.17:0/3258762294 >> 172.21.15.17:0/3258762294 conn(0x7fbc700058f0 msgr2=0x7fbc700043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:48.292 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:48.301+0000 7fbc7dde8640 1 -- 172.21.15.17:0/3258762294 shutdown_connections 2024-03-29T11:28:48.292 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:48.301+0000 7fbc7dde8640 1 -- 172.21.15.17:0/3258762294 wait complete. 2024-03-29T11:28:48.292 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:48.302+0000 7fbc7dde8640 1 librados: shutdown 2024-03-29T11:28:48.310 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:55.118 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:55.248 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.257+0000 7ff657a89640 10 monclient: get_monmap_and_config 2024-03-29T11:28:55.248 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.257+0000 7ff657a89640 10 monclient: build_initial_monmap 2024-03-29T11:28:55.248 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.257+0000 7ff657a89640 10 monclient: monmap: 2024-03-29T11:28:55.248 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:28:55.249 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:28:55.249 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:28:55.258762+0000 2024-03-29T11:28:55.249 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:28:55.258762+0000 2024-03-29T11:28:55.249 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:55.249 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:55.249 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:28:55.249 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:28:55.249 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:28:55.249 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:55.249 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.258+0000 7ff657a89640 1 Processor -- start 2024-03-29T11:28:55.250 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.258+0000 7ff657a89640 1 -- start start 2024-03-29T11:28:55.250 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.258+0000 7ff657a89640 10 monclient: init 2024-03-29T11:28:55.250 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.259+0000 7ff657a89640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:55.250 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.259+0000 7ff657a89640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:28:55.250 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.259+0000 7ff657a89640 10 monclient(hunting): picked mon.noname-b con 0x7ff6500a9750 addr v1:172.21.15.17:6790/0 2024-03-29T11:28:55.250 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.259+0000 7ff657a89640 10 monclient(hunting): picked mon.noname-a con 0x7ff65013e1e0 addr v1:172.21.15.17:6789/0 2024-03-29T11:28:55.250 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.259+0000 7ff657a89640 10 monclient(hunting): picked mon.noname-c con 0x7ff650071c30 addr v1:172.21.15.94:6789/0 2024-03-29T11:28:55.251 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.259+0000 7ff657a89640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6500ab300 con 0x7ff65013e1e0 2024-03-29T11:28:55.251 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.259+0000 7ff657a89640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6500ac2f0 con 0x7ff650071c30 2024-03-29T11:28:55.251 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.259+0000 7ff657a89640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6500ad8d0 con 0x7ff6500a9750 2024-03-29T11:28:55.251 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.259+0000 7ff657a89640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:55.251 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.259+0000 7ff657a89640 10 monclient(hunting): authenticate will time out at 1441.464232s 2024-03-29T11:28:55.251 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.259+0000 7ff656026640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7ff650071c30 0x7ff6500ace20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:53624/0 (socket says 172.21.15.17:53624) 2024-03-29T11:28:55.251 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.259+0000 7ff656026640 1 -- 172.21.15.17:0/4140142509 learned_addr learned my addr 172.21.15.17:0/4140142509 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:28:55.257 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.262+0000 7ff654823640 1 -- 172.21.15.17:0/4140142509 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2061924858 0 0) 0x7ff6500ac2f0 con 0x7ff650071c30 2024-03-29T11:28:55.257 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.262+0000 7ff654823640 10 monclient(hunting): _negotiate 2024-03-29T11:28:55.257 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.262+0000 7ff654823640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:55.257 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.262+0000 7ff654823640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:55.258 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.262+0000 7ff654823640 10 monclient(hunting): my global_id is 4588 2024-03-29T11:28:55.258 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.262+0000 7ff654823640 1 -- 172.21.15.17:0/4140142509 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff63c003520 con 0x7ff650071c30 2024-03-29T11:28:55.258 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.262+0000 7ff654823640 1 -- 172.21.15.17:0/4140142509 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2912855133 0 0) 0x7ff6500ad8d0 con 0x7ff6500a9750 2024-03-29T11:28:55.258 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.262+0000 7ff654823640 10 monclient(hunting): _negotiate 2024-03-29T11:28:55.258 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.262+0000 7ff654823640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:55.258 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.262+0000 7ff654823640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:55.258 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.262+0000 7ff654823640 10 monclient(hunting): my global_id is 4637 2024-03-29T11:28:55.258 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.262+0000 7ff654823640 1 -- 172.21.15.17:0/4140142509 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6500ac2f0 con 0x7ff6500a9750 2024-03-29T11:28:55.258 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.262+0000 7ff654823640 1 -- 172.21.15.17:0/4140142509 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3070121479 0 0) 0x7ff6500ac2f0 con 0x7ff6500a9750 2024-03-29T11:28:55.258 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.262+0000 7ff654823640 1 -- 172.21.15.17:0/4140142509 >> v1:172.21.15.94:6789/0 conn(0x7ff650071c30 legacy=0x7ff6500ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:55.258 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.262+0000 7ff654823640 1 -- 172.21.15.17:0/4140142509 >> v1:172.21.15.17:6789/0 conn(0x7ff65013e1e0 legacy=0x7ff6500ac710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:55.258 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.262+0000 7ff654823640 10 monclient: _finish_hunting 0 2024-03-29T11:28:55.258 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.262+0000 7ff654823640 1 monclient: found mon.noname-b 2024-03-29T11:28:55.259 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.262+0000 7ff654823640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:55.259 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.262+0000 7ff654823640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.17:6790/0 2024-03-29T11:28:55.259 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.262+0000 7ff654823640 1 -- 172.21.15.17:0/4140142509 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6500ada60 con 0x7ff6500a9750 2024-03-29T11:28:55.259 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.262+0000 7ff654823640 10 monclient: _finish_auth 0 2024-03-29T11:28:55.259 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.262+0000 7ff654823640 10 monclient: _check_auth_tickets 2024-03-29T11:28:55.259 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.262+0000 7ff654823640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:55.259 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.262+0000 7ff654823640 1 -- 172.21.15.17:0/4140142509 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7ff64c001520 con 0x7ff6500a9750 2024-03-29T11:28:55.259 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.262+0000 7ff654823640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:55.259 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.262+0000 7ff654823640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:55.259 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.262+0000 7ff654823640 10 monclient: dump: 2024-03-29T11:28:55.259 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:28:55.259 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:28:55.259 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:55.259 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:55.259 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:55.260 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:55.260 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:28:55.260 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:28:55.260 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:28:55.260 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:55.260 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.262+0000 7ff657a89640 5 monclient: authenticate success, global_id 4637 2024-03-29T11:28:55.260 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.262+0000 7ff657a89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:55.260 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.263+0000 7ff654823640 1 -- 172.21.15.17:0/4140142509 <== mon.2 v1:172.21.15.17:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff64c001680 con 0x7ff6500a9750 2024-03-29T11:28:55.260 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.263+0000 7ff654823640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:55.260 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.263+0000 7ff654823640 1 -- 172.21.15.17:0/4140142509 <== mon.2 v1:172.21.15.17:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7ff64c001a90 con 0x7ff6500a9750 2024-03-29T11:28:55.260 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.263+0000 7ff657a89640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:55.260 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.263+0000 7ff657a89640 4 set_mon_vals no callback set 2024-03-29T11:28:55.260 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.263+0000 7ff654823640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:55.260 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.263+0000 7ff654823640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:55.261 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.263+0000 7ff654823640 10 monclient: dump: 2024-03-29T11:28:55.261 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:28:55.261 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:28:55.261 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:55.261 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:55.261 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:55.261 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:55.261 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:28:55.261 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:28:55.261 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:28:55.261 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:55.261 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.263+0000 7ff657a89640 10 monclient: shutdown 2024-03-29T11:28:55.261 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.263+0000 7ff657a89640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:55.261 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.263+0000 7ff657a89640 1 -- 172.21.15.17:0/4140142509 >> v1:172.21.15.17:6790/0 conn(0x7ff6500a9750 legacy=0x7ff65013dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:55.262 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.264+0000 7ff657a89640 1 -- 172.21.15.17:0/4140142509 shutdown_connections 2024-03-29T11:28:55.262 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.264+0000 7ff657a89640 1 -- 172.21.15.17:0/4140142509 >> 172.21.15.17:0/4140142509 conn(0x7ff65006b660 msgr2=0x7ff65006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:55.262 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.264+0000 7ff657a89640 1 -- 172.21.15.17:0/4140142509 shutdown_connections 2024-03-29T11:28:55.262 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.264+0000 7ff657a89640 1 -- 172.21.15.17:0/4140142509 wait complete. 2024-03-29T11:28:55.262 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.265+0000 7ff657a89640 10 monclient: build_initial_monmap 2024-03-29T11:28:55.262 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.265+0000 7ff657a89640 10 monclient: monmap: 2024-03-29T11:28:55.262 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:28:55.262 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:55.262 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:28:55.262 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:28:55.262 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:55.262 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:55.262 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:28:55.262 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:28:55.262 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:28:55.263 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:55.263 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.265+0000 7ff657a89640 1 librados: starting msgr at 2024-03-29T11:28:55.263 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.265+0000 7ff657a89640 1 librados: starting objecter 2024-03-29T11:28:55.263 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.265+0000 7ff657a89640 1 Processor -- start 2024-03-29T11:28:55.263 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.265+0000 7ff657a89640 1 -- start start 2024-03-29T11:28:55.263 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.265+0000 7ff657a89640 1 librados: setting wanted keys 2024-03-29T11:28:55.263 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.265+0000 7ff657a89640 1 librados: calling monclient init 2024-03-29T11:28:55.263 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.265+0000 7ff657a89640 10 monclient: init 2024-03-29T11:28:55.263 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.266+0000 7ff657a89640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:55.263 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.266+0000 7ff657a89640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:28:55.263 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.266+0000 7ff657a89640 10 monclient(hunting): picked mon.noname-c con 0x7ff650071c30 addr v1:172.21.15.17:6790/0 2024-03-29T11:28:55.263 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.266+0000 7ff655825640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7ff650071c30 0x7ff6501e7980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:35080/0 (socket says 172.21.15.17:35080) 2024-03-29T11:28:55.263 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.266+0000 7ff657a89640 10 monclient(hunting): picked mon.noname-b con 0x7ff65013e1e0 addr v1:172.21.15.94:6789/0 2024-03-29T11:28:55.263 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.267+0000 7ff657a89640 10 monclient(hunting): picked mon.noname-a con 0x7ff6501427c0 addr v1:172.21.15.17:6789/0 2024-03-29T11:28:55.263 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.267+0000 7ff657a89640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff650140980 con 0x7ff6501427c0 2024-03-29T11:28:55.264 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.267+0000 7ff657a89640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff650141de0 con 0x7ff65013e1e0 2024-03-29T11:28:55.264 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.267+0000 7ff657a89640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6501420c0 con 0x7ff650071c30 2024-03-29T11:28:55.264 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.267+0000 7ff657a89640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:55.264 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.267+0000 7ff657a89640 10 monclient(hunting): authenticate will time out at 1441.472111s 2024-03-29T11:28:55.264 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.267+0000 7ff655825640 1 -- 172.21.15.17:0/240987230 learned_addr learned my addr 172.21.15.17:0/240987230 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:28:55.264 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.267+0000 7ff6467fc640 10 client.?.objecter ms_handle_connect 0x7ff650071c30 2024-03-29T11:28:55.264 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.267+0000 7ff6467fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:55.265 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.267+0000 7ff6467fc640 1 -- 172.21.15.17:0/240987230 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 531591614 0 0) 0x7ff6501420c0 con 0x7ff650071c30 2024-03-29T11:28:55.266 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.267+0000 7ff6467fc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:55.266 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.267+0000 7ff6467fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:55.266 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.267+0000 7ff6467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:55.266 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.267+0000 7ff6467fc640 10 monclient(hunting): my global_id is 4640 2024-03-29T11:28:55.266 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.267+0000 7ff6467fc640 1 -- 172.21.15.17:0/240987230 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff634004eb0 con 0x7ff650071c30 2024-03-29T11:28:55.266 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.268+0000 7ff6467fc640 10 client.?.objecter ms_handle_connect 0x7ff65013e1e0 2024-03-29T11:28:55.266 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.268+0000 7ff6467fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:55.266 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.268+0000 7ff6467fc640 1 -- 172.21.15.17:0/240987230 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3238759535 0 0) 0x7ff634004eb0 con 0x7ff650071c30 2024-03-29T11:28:55.266 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.268+0000 7ff6467fc640 1 -- 172.21.15.17:0/240987230 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6501420c0 con 0x7ff650071c30 2024-03-29T11:28:55.266 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.268+0000 7ff6467fc640 1 -- 172.21.15.17:0/240987230 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7ff64c001990 con 0x7ff650071c30 2024-03-29T11:28:55.266 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.268+0000 7ff6467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:55.266 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.268+0000 7ff6467fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:55.266 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.268+0000 7ff6467fc640 10 monclient(hunting): dump: 2024-03-29T11:28:55.267 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:28:55.267 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:28:55.267 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:55.267 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:55.267 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:55.267 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:55.267 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:28:55.267 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:28:55.267 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:28:55.267 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:55.267 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.268+0000 7ff6467fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:55.267 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.268+0000 7ff6467fc640 1 -- 172.21.15.17:0/240987230 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2099576914 0 0) 0x7ff650141de0 con 0x7ff65013e1e0 2024-03-29T11:28:55.267 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.268+0000 7ff6467fc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:55.267 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.268+0000 7ff6467fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:55.267 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.268+0000 7ff6467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:55.268 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.268+0000 7ff6467fc640 10 monclient(hunting): my global_id is 4591 2024-03-29T11:28:55.268 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.268+0000 7ff6467fc640 1 -- 172.21.15.17:0/240987230 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff634004eb0 con 0x7ff65013e1e0 2024-03-29T11:28:55.268 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.268+0000 7ff6467fc640 1 -- 172.21.15.17:0/240987230 <== mon.2 v1:172.21.15.17:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 778587000 0 0) 0x7ff6501420c0 con 0x7ff650071c30 2024-03-29T11:28:55.268 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.268+0000 7ff6467fc640 1 -- 172.21.15.17:0/240987230 >> v1:172.21.15.94:6789/0 conn(0x7ff65013e1e0 legacy=0x7ff6501ce820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:55.268 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.268+0000 7ff6467fc640 1 -- 172.21.15.17:0/240987230 >> v1:172.21.15.17:6789/0 conn(0x7ff6501427c0 legacy=0x7ff65006da60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:55.268 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.269+0000 7ff6467fc640 10 monclient: _finish_hunting 0 2024-03-29T11:28:55.268 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.269+0000 7ff6467fc640 1 monclient: found mon.c 2024-03-29T11:28:55.268 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.269+0000 7ff6467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:55.268 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.269+0000 7ff6467fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:28:55.268 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.269+0000 7ff6467fc640 1 -- 172.21.15.17:0/240987230 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6501423a0 con 0x7ff650071c30 2024-03-29T11:28:55.268 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.269+0000 7ff6467fc640 10 monclient: _finish_auth 0 2024-03-29T11:28:55.268 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.269+0000 7ff6467fc640 10 monclient: _check_auth_tickets 2024-03-29T11:28:55.268 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.269+0000 7ff6467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:55.268 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.269+0000 7ff657a89640 5 monclient: authenticate success, global_id 4640 2024-03-29T11:28:55.269 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.269+0000 7ff657a89640 10 monclient: _renew_subs 2024-03-29T11:28:55.269 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.269+0000 7ff657a89640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:28:55.269 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.269+0000 7ff657a89640 1 -- 172.21.15.17:0/240987230 --> v1:172.21.15.17:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6501e8090 con 0x7ff650071c30 2024-03-29T11:28:55.269 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.269+0000 7ff657a89640 10 client.4640.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:55.269 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.269+0000 7ff657a89640 10 monclient: _renew_subs 2024-03-29T11:28:55.269 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.269+0000 7ff657a89640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:28:55.269 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.269+0000 7ff657a89640 1 -- 172.21.15.17:0/240987230 --> v1:172.21.15.17:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6501e8570 con 0x7ff650071c30 2024-03-29T11:28:55.269 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.269+0000 7ff657a89640 1 librados: init done 2024-03-29T11:28:55.269 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.270+0000 7ff6467fc640 1 -- 172.21.15.17:0/240987230 <== mon.2 v1:172.21.15.17:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff64c0038a0 con 0x7ff650071c30 2024-03-29T11:28:55.269 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.270+0000 7ff6467fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:55.269 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.270+0000 7ff647fff640 4 set_mon_vals no callback set 2024-03-29T11:28:55.269 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.270+0000 7ff6467fc640 1 -- 172.21.15.17:0/240987230 <== mon.2 v1:172.21.15.17:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7ff64c003c30 con 0x7ff650071c30 2024-03-29T11:28:55.269 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.270+0000 7ff6467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:55.269 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.270+0000 7ff6467fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:55.269 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.270+0000 7ff6467fc640 10 monclient: dump: 2024-03-29T11:28:55.270 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:28:55.270 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:28:55.270 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:55.270 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:28:55.270 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:55.270 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:28:55.270 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:28:55.270 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:28:55.270 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:28:55.270 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:28:55.270 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.271+0000 7ff657a89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:55.270 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.271+0000 7ff657a89640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:55.270 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.271+0000 7ff657a89640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:28:55.270 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.271+0000 7ff657a89640 1 -- 172.21.15.17:0/240987230 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff638005080 con 0x7ff650071c30 2024-03-29T11:28:55.270 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.271+0000 7ff6467fc640 1 -- 172.21.15.17:0/240987230 <== mon.2 v1:172.21.15.17:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7ff64c003d90 con 0x7ff650071c30 2024-03-29T11:28:55.271 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.272+0000 7ff6467fc640 1 -- 172.21.15.17:0/240987230 <== mon.2 v1:172.21.15.17:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1968932881 0 0) 0x7ff64c084b40 con 0x7ff650071c30 2024-03-29T11:28:55.271 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.272+0000 7ff655024640 10 monclient: get_auth_request con 0x7ff63406ddd0 auth_method 0 2024-03-29T11:28:55.271 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.272+0000 7ff6467fc640 10 client.4640.objecter ms_dispatch 0x7ff650000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:55.271 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.272+0000 7ff6467fc640 3 client.4640.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:55.271 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.272+0000 7ff6467fc640 3 client.4640.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:55.271 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.272+0000 7ff6467fc640 20 client.4640.objecter dump_active .. 0 homeless 2024-03-29T11:28:55.271 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.273+0000 7ff6467fc640 10 client.4640.objecter ms_handle_connect 0x7ff63406ddd0 2024-03-29T11:28:55.272 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.281+0000 7ff6467fc640 1 -- 172.21.15.17:0/240987230 <== mon.2 v1:172.21.15.17:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff64c050420 con 0x7ff650071c30 2024-03-29T11:28:55.272 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.281+0000 7ff6467fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:55.272 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.281+0000 7ff6467fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:55.585 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.594+0000 7ff657a89640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:55.585 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.594+0000 7ff657a89640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:55.585 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.594+0000 7ff657a89640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:28:55.585 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.594+0000 7ff657a89640 1 -- 172.21.15.17:0/240987230 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff638005330 con 0x7ff650071c30 2024-03-29T11:28:55.586 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.595+0000 7ff6467fc640 1 -- 172.21.15.17:0/240987230 <== mon.2 v1:172.21.15.17:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 591375189) 0x7ff64c052350 con 0x7ff650071c30 2024-03-29T11:28:55.586 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.595+0000 7ff6467fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:55.586 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.595+0000 7ff6467fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:55.586 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:28:55.586 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:18:58.393037+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:58.393037+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:55.590 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.599+0000 7ff657a89640 10 librados: watch_flush enter 2024-03-29T11:28:55.590 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.599+0000 7ff657a89640 10 librados: watch_flush exit 2024-03-29T11:28:55.590 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.599+0000 7ff657a89640 20 client.4640.objecter shutdown clearing up homeless session... 2024-03-29T11:28:55.590 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.599+0000 7ff657a89640 10 client.4640.objecter successfully canceled tick 2024-03-29T11:28:55.590 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.600+0000 7ff657a89640 1 -- 172.21.15.17:0/240987230 >> v1:172.21.15.17:6816/3236185052 conn(0x7ff63406ddd0 legacy=0x7ff634070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:55.590 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.600+0000 7ff657a89640 10 monclient: shutdown 2024-03-29T11:28:55.590 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.600+0000 7ff657a89640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:55.590 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.600+0000 7ff657a89640 1 -- 172.21.15.17:0/240987230 >> v1:172.21.15.17:6790/0 conn(0x7ff650071c30 legacy=0x7ff6501e7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:55.591 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.600+0000 7ff657a89640 1 -- 172.21.15.17:0/240987230 shutdown_connections 2024-03-29T11:28:55.591 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.600+0000 7ff657a89640 1 -- 172.21.15.17:0/240987230 >> 172.21.15.17:0/240987230 conn(0x7ff65006b660 msgr2=0x7ff65013e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:55.591 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.600+0000 7ff657a89640 1 -- 172.21.15.17:0/240987230 shutdown_connections 2024-03-29T11:28:55.591 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.601+0000 7ff657a89640 1 -- 172.21.15.17:0/240987230 wait complete. 2024-03-29T11:28:55.591 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:28:55.601+0000 7ff657a89640 1 librados: shutdown 2024-03-29T11:28:55.604 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:02.411 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:02.545 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.554+0000 7f7d63385640 10 monclient: get_monmap_and_config 2024-03-29T11:29:02.545 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.554+0000 7f7d63385640 10 monclient: build_initial_monmap 2024-03-29T11:29:02.545 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.554+0000 7f7d63385640 10 monclient: monmap: 2024-03-29T11:29:02.545 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:29:02.545 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:29:02.545 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:29:02.555368+0000 2024-03-29T11:29:02.545 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:29:02.555368+0000 2024-03-29T11:29:02.545 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:02.546 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:02.546 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:29:02.546 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:29:02.546 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:29:02.546 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:02.547 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.555+0000 7f7d63385640 1 Processor -- start 2024-03-29T11:29:02.547 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.555+0000 7f7d63385640 1 -- start start 2024-03-29T11:29:02.547 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.555+0000 7f7d63385640 10 monclient: init 2024-03-29T11:29:02.547 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.556+0000 7f7d63385640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:02.547 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.556+0000 7f7d63385640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:29:02.547 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.556+0000 7f7d61121640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f7d5c1457f0 0x7f7d5c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:54940/0 (socket says 172.21.15.17:54940) 2024-03-29T11:29:02.547 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.556+0000 7f7d61121640 1 -- 172.21.15.17:0/3521441803 learned_addr learned my addr 172.21.15.17:0/3521441803 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:29:02.547 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.556+0000 7f7d63385640 10 monclient(hunting): picked mon.noname-b con 0x7f7d5c1457f0 addr v1:172.21.15.17:6790/0 2024-03-29T11:29:02.547 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.556+0000 7f7d63385640 10 monclient(hunting): picked mon.noname-c con 0x7f7d5c13f650 addr v1:172.21.15.94:6789/0 2024-03-29T11:29:02.548 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.556+0000 7f7d63385640 10 monclient(hunting): picked mon.noname-a con 0x7f7d5c0a9d70 addr v1:172.21.15.17:6789/0 2024-03-29T11:29:02.548 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.556+0000 7f7d63385640 1 -- 172.21.15.17:0/3521441803 --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d5c0aaef0 con 0x7f7d5c0a9d70 2024-03-29T11:29:02.548 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.556+0000 7f7d63385640 1 -- 172.21.15.17:0/3521441803 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d5c145bd0 con 0x7f7d5c13f650 2024-03-29T11:29:02.548 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.556+0000 7f7d63385640 1 -- 172.21.15.17:0/3521441803 --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d5c145eb0 con 0x7f7d5c1457f0 2024-03-29T11:29:02.548 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.556+0000 7f7d63385640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:02.548 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.556+0000 7f7d63385640 10 monclient(hunting): authenticate will time out at 1448.761827s 2024-03-29T11:29:02.548 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.557+0000 7f7d53fff640 1 -- 172.21.15.17:0/3521441803 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1437612217 0 0) 0x7f7d5c145eb0 con 0x7f7d5c1457f0 2024-03-29T11:29:02.548 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.557+0000 7f7d53fff640 10 monclient(hunting): _negotiate 2024-03-29T11:29:02.548 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.557+0000 7f7d53fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:02.549 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.557+0000 7f7d53fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:02.549 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.557+0000 7f7d53fff640 10 monclient(hunting): my global_id is 4643 2024-03-29T11:29:02.549 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.557+0000 7f7d53fff640 1 -- 172.21.15.17:0/3521441803 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d48003520 con 0x7f7d5c1457f0 2024-03-29T11:29:02.549 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.557+0000 7f7d53fff640 1 -- 172.21.15.17:0/3521441803 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 489137544 0 0) 0x7f7d5c145bd0 con 0x7f7d5c13f650 2024-03-29T11:29:02.549 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.557+0000 7f7d53fff640 10 monclient(hunting): _negotiate 2024-03-29T11:29:02.549 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.557+0000 7f7d53fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:02.549 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.557+0000 7f7d53fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:02.549 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.557+0000 7f7d53fff640 10 monclient(hunting): my global_id is 4594 2024-03-29T11:29:02.549 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.557+0000 7f7d53fff640 1 -- 172.21.15.17:0/3521441803 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d5c145eb0 con 0x7f7d5c13f650 2024-03-29T11:29:02.549 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.557+0000 7f7d53fff640 1 -- 172.21.15.17:0/3521441803 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 46704655 0 0) 0x7f7d48003520 con 0x7f7d5c1457f0 2024-03-29T11:29:02.549 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.557+0000 7f7d53fff640 1 -- 172.21.15.17:0/3521441803 >> v1:172.21.15.94:6789/0 conn(0x7f7d5c13f650 legacy=0x7f7d5c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:02.549 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.557+0000 7f7d53fff640 1 -- 172.21.15.17:0/3521441803 >> v1:172.21.15.17:6789/0 conn(0x7f7d5c0a9d70 legacy=0x7f7d5c14d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:02.552 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.560+0000 7f7d53fff640 10 monclient: _finish_hunting 0 2024-03-29T11:29:02.553 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.560+0000 7f7d53fff640 1 monclient: found mon.noname-b 2024-03-29T11:29:02.553 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.560+0000 7f7d53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:02.553 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.560+0000 7f7d53fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.17:6790/0 2024-03-29T11:29:02.553 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.560+0000 7f7d53fff640 1 -- 172.21.15.17:0/3521441803 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d5c146340 con 0x7f7d5c1457f0 2024-03-29T11:29:02.553 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.560+0000 7f7d53fff640 10 monclient: _finish_auth 0 2024-03-29T11:29:02.553 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.560+0000 7f7d53fff640 10 monclient: _check_auth_tickets 2024-03-29T11:29:02.553 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.560+0000 7f7d53fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:02.553 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.560+0000 7f7d53fff640 1 -- 172.21.15.17:0/3521441803 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f7d58001690 con 0x7f7d5c1457f0 2024-03-29T11:29:02.553 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.560+0000 7f7d53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:02.553 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.560+0000 7f7d53fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:02.553 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.560+0000 7f7d53fff640 10 monclient: dump: 2024-03-29T11:29:02.553 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:29:02.553 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:29:02.554 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:02.554 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:02.554 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:02.554 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:02.554 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:29:02.554 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:29:02.554 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:29:02.554 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:02.554 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.560+0000 7f7d63385640 5 monclient: authenticate success, global_id 4643 2024-03-29T11:29:02.554 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.560+0000 7f7d63385640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:02.554 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.560+0000 7f7d53fff640 1 -- 172.21.15.17:0/3521441803 <== mon.2 v1:172.21.15.17:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d580017f0 con 0x7f7d5c1457f0 2024-03-29T11:29:02.554 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.560+0000 7f7d53fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:02.554 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.560+0000 7f7d53fff640 1 -- 172.21.15.17:0/3521441803 <== mon.2 v1:172.21.15.17:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f7d58001c00 con 0x7f7d5c1457f0 2024-03-29T11:29:02.554 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.560+0000 7f7d63385640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:02.555 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.560+0000 7f7d63385640 4 set_mon_vals no callback set 2024-03-29T11:29:02.555 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.560+0000 7f7d53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:02.555 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.560+0000 7f7d53fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:02.555 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.560+0000 7f7d53fff640 10 monclient: dump: 2024-03-29T11:29:02.555 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:29:02.555 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:29:02.555 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:02.555 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:02.555 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:02.555 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:02.555 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:29:02.555 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:29:02.555 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:29:02.555 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:02.555 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.561+0000 7f7d63385640 10 monclient: shutdown 2024-03-29T11:29:02.556 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.561+0000 7f7d63385640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:02.556 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.561+0000 7f7d63385640 1 -- 172.21.15.17:0/3521441803 >> v1:172.21.15.17:6790/0 conn(0x7f7d5c1457f0 legacy=0x7f7d5c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:02.556 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.562+0000 7f7d63385640 1 -- 172.21.15.17:0/3521441803 shutdown_connections 2024-03-29T11:29:02.556 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.562+0000 7f7d63385640 1 -- 172.21.15.17:0/3521441803 >> 172.21.15.17:0/3521441803 conn(0x7f7d5c066a00 msgr2=0x7f7d5c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:02.556 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.562+0000 7f7d63385640 1 -- 172.21.15.17:0/3521441803 shutdown_connections 2024-03-29T11:29:02.562 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.562+0000 7f7d63385640 1 -- 172.21.15.17:0/3521441803 wait complete. 2024-03-29T11:29:02.562 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.563+0000 7f7d63385640 10 monclient: build_initial_monmap 2024-03-29T11:29:02.562 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.563+0000 7f7d63385640 10 monclient: monmap: 2024-03-29T11:29:02.563 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:29:02.563 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:02.563 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:29:02.563 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:29:02.563 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:02.563 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:02.563 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:29:02.563 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:29:02.563 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:29:02.563 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:02.563 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.563+0000 7f7d63385640 1 librados: starting msgr at 2024-03-29T11:29:02.564 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.563+0000 7f7d63385640 1 librados: starting objecter 2024-03-29T11:29:02.564 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.563+0000 7f7d63385640 1 Processor -- start 2024-03-29T11:29:02.564 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.563+0000 7f7d63385640 1 -- start start 2024-03-29T11:29:02.564 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.563+0000 7f7d63385640 1 librados: setting wanted keys 2024-03-29T11:29:02.564 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.563+0000 7f7d63385640 1 librados: calling monclient init 2024-03-29T11:29:02.564 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.563+0000 7f7d63385640 10 monclient: init 2024-03-29T11:29:02.564 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.564+0000 7f7d63385640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:02.564 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.564+0000 7f7d63385640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:29:02.564 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.564+0000 7f7d63385640 10 monclient(hunting): picked mon.noname-b con 0x7f7d5c0a9d70 addr v1:172.21.15.94:6789/0 2024-03-29T11:29:02.564 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.564+0000 7f7d63385640 10 monclient(hunting): picked mon.noname-c con 0x7f7d5c13f650 addr v1:172.21.15.17:6790/0 2024-03-29T11:29:02.564 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.564+0000 7f7d63385640 10 monclient(hunting): picked mon.noname-a con 0x7f7d5c1457f0 addr v1:172.21.15.17:6789/0 2024-03-29T11:29:02.565 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.564+0000 7f7d63385640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d5c13a840 con 0x7f7d5c1457f0 2024-03-29T11:29:02.565 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.564+0000 7f7d63385640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d5c071a70 con 0x7f7d5c0a9d70 2024-03-29T11:29:02.565 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.564+0000 7f7d63385640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d5c070200 con 0x7f7d5c13f650 2024-03-29T11:29:02.565 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.564+0000 7f7d63385640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:02.565 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.564+0000 7f7d63385640 10 monclient(hunting): authenticate will time out at 1448.769430s 2024-03-29T11:29:02.565 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.564+0000 7f7d60920640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f7d5c13f650 0x7f7d5c1dec30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:56462/0 (socket says 172.21.15.17:56462) 2024-03-29T11:29:02.565 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.564+0000 7f7d60920640 1 -- 172.21.15.17:0/3941094598 learned_addr learned my addr 172.21.15.17:0/3941094598 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:29:02.565 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.564+0000 7f7d51ffb640 10 client.?.objecter ms_handle_connect 0x7f7d5c13f650 2024-03-29T11:29:02.565 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.564+0000 7f7d51ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:02.565 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.564+0000 7f7d51ffb640 10 client.?.objecter ms_handle_connect 0x7f7d5c0a9d70 2024-03-29T11:29:02.565 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.564+0000 7f7d51ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:02.565 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.564+0000 7f7d51ffb640 1 -- 172.21.15.17:0/3941094598 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2999707703 0 0) 0x7f7d5c070200 con 0x7f7d5c13f650 2024-03-29T11:29:02.565 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.564+0000 7f7d51ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:29:02.565 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.564+0000 7f7d51ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:02.566 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.564+0000 7f7d51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:02.566 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.564+0000 7f7d51ffb640 10 monclient(hunting): my global_id is 4646 2024-03-29T11:29:02.566 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.564+0000 7f7d51ffb640 1 -- 172.21.15.17:0/3941094598 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d40004eb0 con 0x7f7d5c13f650 2024-03-29T11:29:02.566 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.565+0000 7f7d51ffb640 1 -- 172.21.15.17:0/3941094598 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2711735279 0 0) 0x7f7d5c071a70 con 0x7f7d5c0a9d70 2024-03-29T11:29:02.566 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.565+0000 7f7d51ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:29:02.566 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.565+0000 7f7d51ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:02.566 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.565+0000 7f7d51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:02.566 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.565+0000 7f7d51ffb640 10 monclient(hunting): my global_id is 4597 2024-03-29T11:29:02.566 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.565+0000 7f7d51ffb640 1 -- 172.21.15.17:0/3941094598 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d5c070200 con 0x7f7d5c0a9d70 2024-03-29T11:29:02.566 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.565+0000 7f7d51ffb640 1 -- 172.21.15.17:0/3941094598 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2872587261 0 0) 0x7f7d40004eb0 con 0x7f7d5c13f650 2024-03-29T11:29:02.566 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.565+0000 7f7d51ffb640 1 -- 172.21.15.17:0/3941094598 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d5c071a70 con 0x7f7d5c13f650 2024-03-29T11:29:02.566 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.565+0000 7f7d51ffb640 1 -- 172.21.15.17:0/3941094598 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f7d4c001a10 con 0x7f7d5c13f650 2024-03-29T11:29:02.566 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.565+0000 7f7d51ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:02.566 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.565+0000 7f7d51ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:02.566 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.565+0000 7f7d51ffb640 10 monclient(hunting): dump: 2024-03-29T11:29:02.567 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:29:02.567 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:29:02.567 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:02.567 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:02.567 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:02.567 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:02.567 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:29:02.567 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:29:02.567 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:29:02.567 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:02.567 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.565+0000 7f7d51ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:02.567 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.566+0000 7f7d51ffb640 1 -- 172.21.15.17:0/3941094598 <== mon.2 v1:172.21.15.17:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2915004946 0 0) 0x7f7d5c071a70 con 0x7f7d5c13f650 2024-03-29T11:29:02.567 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.567+0000 7f7d51ffb640 1 -- 172.21.15.17:0/3941094598 >> v1:172.21.15.94:6789/0 conn(0x7f7d5c0a9d70 legacy=0x7f7d5c141ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:02.567 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.567+0000 7f7d51ffb640 1 -- 172.21.15.17:0/3941094598 >> v1:172.21.15.17:6789/0 conn(0x7f7d5c1457f0 legacy=0x7f7d5c13a130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:29:02.567 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.567+0000 7f7d51ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:29:02.568 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.567+0000 7f7d51ffb640 1 monclient: found mon.c 2024-03-29T11:29:02.568 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.567+0000 7f7d51ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:02.568 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.567+0000 7f7d51ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:29:02.568 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.567+0000 7f7d51ffb640 1 -- 172.21.15.17:0/3941094598 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d5c06fbb0 con 0x7f7d5c13f650 2024-03-29T11:29:02.568 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.567+0000 7f7d51ffb640 10 monclient: _finish_auth 0 2024-03-29T11:29:02.568 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.567+0000 7f7d51ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:29:02.568 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.567+0000 7f7d51ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:02.568 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.567+0000 7f7d63385640 5 monclient: authenticate success, global_id 4646 2024-03-29T11:29:02.568 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.567+0000 7f7d63385640 10 monclient: _renew_subs 2024-03-29T11:29:02.568 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.567+0000 7f7d63385640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:29:02.568 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.567+0000 7f7d63385640 1 -- 172.21.15.17:0/3941094598 --> v1:172.21.15.17:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d5c06fe20 con 0x7f7d5c13f650 2024-03-29T11:29:02.568 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.568+0000 7f7d63385640 10 client.4646.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:02.568 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.568+0000 7f7d63385640 10 monclient: _renew_subs 2024-03-29T11:29:02.568 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.568+0000 7f7d63385640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:29:02.568 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.568+0000 7f7d63385640 1 -- 172.21.15.17:0/3941094598 --> v1:172.21.15.17:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d5c070900 con 0x7f7d5c13f650 2024-03-29T11:29:02.569 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.568+0000 7f7d63385640 1 librados: init done 2024-03-29T11:29:02.569 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.569+0000 7f7d51ffb640 1 -- 172.21.15.17:0/3941094598 <== mon.2 v1:172.21.15.17:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d4c001c50 con 0x7f7d5c13f650 2024-03-29T11:29:02.569 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.569+0000 7f7d51ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:02.569 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.569+0000 7f7d51ffb640 1 -- 172.21.15.17:0/3941094598 <== mon.2 v1:172.21.15.17:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f7d4c003a60 con 0x7f7d5c13f650 2024-03-29T11:29:02.569 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.569+0000 7f7d51ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:02.569 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.569+0000 7f7d51ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:02.569 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.569+0000 7f7d51ffb640 10 monclient: dump: 2024-03-29T11:29:02.569 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:29:02.569 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:29:02.569 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:02.569 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:02.569 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:02.569 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:02.569 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:29:02.570 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:29:02.570 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:29:02.570 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:02.570 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.569+0000 7f7d537fe640 4 set_mon_vals no callback set 2024-03-29T11:29:02.570 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.570+0000 7f7d51ffb640 1 -- 172.21.15.17:0/3941094598 <== mon.2 v1:172.21.15.17:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f7d4c003bc0 con 0x7f7d5c13f650 2024-03-29T11:29:02.570 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.571+0000 7f7d63385640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:02.570 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.571+0000 7f7d51ffb640 1 -- 172.21.15.17:0/3941094598 <== mon.2 v1:172.21.15.17:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3682049282 0 0) 0x7f7d4c084af0 con 0x7f7d5c13f650 2024-03-29T11:29:02.570 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.571+0000 7f7d51ffb640 10 client.4646.objecter ms_dispatch 0x7f7d5c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:29:02.570 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.571+0000 7f7d51ffb640 3 client.4646.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:29:02.570 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.571+0000 7f7d51ffb640 3 client.4646.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:29:02.570 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.571+0000 7f7d51ffb640 20 client.4646.objecter dump_active .. 0 homeless 2024-03-29T11:29:02.570 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.571+0000 7f7d63385640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:02.570 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.571+0000 7f7d63385640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:29:02.570 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.571+0000 7f7d63385640 1 -- 172.21.15.17:0/3941094598 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d3c005080 con 0x7f7d5c13f650 2024-03-29T11:29:02.570 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.571+0000 7f7d61121640 10 monclient: get_auth_request con 0x7f7d4006da80 auth_method 0 2024-03-29T11:29:02.571 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.572+0000 7f7d51ffb640 10 client.4646.objecter ms_handle_connect 0x7f7d4006da80 2024-03-29T11:29:02.571 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.579+0000 7f7d51ffb640 1 -- 172.21.15.17:0/3941094598 <== mon.2 v1:172.21.15.17:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7d4c0503d0 con 0x7f7d5c13f650 2024-03-29T11:29:02.571 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.580+0000 7f7d51ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:02.571 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.580+0000 7f7d51ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:02.888 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.897+0000 7f7d63385640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:02.888 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.897+0000 7f7d63385640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:02.888 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.897+0000 7f7d63385640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:29:02.888 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.897+0000 7f7d63385640 1 -- 172.21.15.17:0/3941094598 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d3c005330 con 0x7f7d5c13f650 2024-03-29T11:29:02.889 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.898+0000 7f7d51ffb640 1 -- 172.21.15.17:0/3941094598 <== mon.2 v1:172.21.15.17:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 543280943) 0x7f7d4c052300 con 0x7f7d5c13f650 2024-03-29T11:29:02.889 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.898+0000 7f7d51ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:02.889 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.898+0000 7f7d51ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:02.890 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:29:02.890 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":11,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:29:01.208095+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:01.208095+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:02.893 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.902+0000 7f7d63385640 10 librados: watch_flush enter 2024-03-29T11:29:02.893 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.902+0000 7f7d63385640 10 librados: watch_flush exit 2024-03-29T11:29:02.894 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.902+0000 7f7d63385640 20 client.4646.objecter shutdown clearing up homeless session... 2024-03-29T11:29:02.894 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.902+0000 7f7d63385640 10 client.4646.objecter successfully canceled tick 2024-03-29T11:29:02.894 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.902+0000 7f7d63385640 1 -- 172.21.15.17:0/3941094598 >> v1:172.21.15.17:6816/3236185052 conn(0x7f7d4006da80 legacy=0x7f7d4006ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:02.894 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.902+0000 7f7d63385640 10 monclient: shutdown 2024-03-29T11:29:02.894 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.902+0000 7f7d63385640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:02.894 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.902+0000 7f7d63385640 1 -- 172.21.15.17:0/3941094598 >> v1:172.21.15.17:6790/0 conn(0x7f7d5c13f650 legacy=0x7f7d5c1dec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:02.894 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.902+0000 7f7d63385640 1 -- 172.21.15.17:0/3941094598 shutdown_connections 2024-03-29T11:29:02.894 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.902+0000 7f7d63385640 1 -- 172.21.15.17:0/3941094598 >> 172.21.15.17:0/3941094598 conn(0x7f7d5c066a00 msgr2=0x7f7d5c13ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:02.894 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.903+0000 7f7d63385640 1 -- 172.21.15.17:0/3941094598 shutdown_connections 2024-03-29T11:29:02.894 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.903+0000 7f7d63385640 1 -- 172.21.15.17:0/3941094598 wait complete. 2024-03-29T11:29:02.894 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:02.903+0000 7f7d63385640 1 librados: shutdown 2024-03-29T11:29:02.908 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:09.716 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:09.871 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.879+0000 7f328476b640 10 monclient: get_monmap_and_config 2024-03-29T11:29:09.872 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.879+0000 7f328476b640 10 monclient: build_initial_monmap 2024-03-29T11:29:09.872 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.879+0000 7f328476b640 10 monclient: monmap: 2024-03-29T11:29:09.872 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:29:09.872 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:29:09.872 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:29:09.880387+0000 2024-03-29T11:29:09.872 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:29:09.880387+0000 2024-03-29T11:29:09.872 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:09.872 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:09.872 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:29:09.872 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:29:09.872 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:29:09.872 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:09.872 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.880+0000 7f328476b640 1 Processor -- start 2024-03-29T11:29:09.873 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.880+0000 7f328476b640 1 -- start start 2024-03-29T11:29:09.873 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.880+0000 7f328476b640 10 monclient: init 2024-03-29T11:29:09.873 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.881+0000 7f328476b640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:09.873 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.881+0000 7f328476b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:29:09.873 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.881+0000 7f328476b640 10 monclient(hunting): picked mon.noname-a con 0x7f327c0ae250 addr v1:172.21.15.17:6789/0 2024-03-29T11:29:09.873 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.881+0000 7f328476b640 10 monclient(hunting): picked mon.noname-b con 0x7f327c0a9720 addr v1:172.21.15.17:6790/0 2024-03-29T11:29:09.874 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.881+0000 7f328476b640 10 monclient(hunting): picked mon.noname-c con 0x7f327c14bba0 addr v1:172.21.15.94:6789/0 2024-03-29T11:29:09.874 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.881+0000 7f328476b640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f327c0a9b80 con 0x7f327c0ae250 2024-03-29T11:29:09.874 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.881+0000 7f328476b640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f327c0aa7b0 con 0x7f327c14bba0 2024-03-29T11:29:09.874 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.881+0000 7f328476b640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f327c0afbf0 con 0x7f327c0a9720 2024-03-29T11:29:09.874 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.881+0000 7f328476b640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:09.874 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.881+0000 7f328476b640 10 monclient(hunting): authenticate will time out at 1456.086797s 2024-03-29T11:29:09.874 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.883+0000 7f3281d06640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f327c0a9720 0x7f327c13f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:37988/0 (socket says 172.21.15.17:37988) 2024-03-29T11:29:09.874 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.883+0000 7f3281d06640 1 -- 172.21.15.17:0/597614184 learned_addr learned my addr 172.21.15.17:0/597614184 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:29:09.876 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.885+0000 7f3281505640 1 -- 172.21.15.17:0/597614184 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1404169437 0 0) 0x7f327c0afbf0 con 0x7f327c0a9720 2024-03-29T11:29:09.877 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.885+0000 7f3281505640 10 monclient(hunting): _negotiate 2024-03-29T11:29:09.877 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.885+0000 7f3281505640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:09.877 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.885+0000 7f3281505640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:09.877 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.885+0000 7f3281505640 10 monclient(hunting): my global_id is 4649 2024-03-29T11:29:09.877 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.885+0000 7f3281505640 1 -- 172.21.15.17:0/597614184 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3268003520 con 0x7f327c0a9720 2024-03-29T11:29:09.878 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.886+0000 7f3281505640 1 -- 172.21.15.17:0/597614184 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 805804036 0 0) 0x7f327c0aa7b0 con 0x7f327c14bba0 2024-03-29T11:29:09.878 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.886+0000 7f3281505640 10 monclient(hunting): _negotiate 2024-03-29T11:29:09.878 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.886+0000 7f3281505640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:09.878 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.886+0000 7f3281505640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:09.878 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.886+0000 7f3281505640 10 monclient(hunting): my global_id is 4600 2024-03-29T11:29:09.878 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.886+0000 7f3281505640 1 -- 172.21.15.17:0/597614184 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3268003840 con 0x7f327c14bba0 2024-03-29T11:29:09.878 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.886+0000 7f3281505640 1 -- 172.21.15.17:0/597614184 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3566250817 0 0) 0x7f3268003520 con 0x7f327c0a9720 2024-03-29T11:29:09.878 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.886+0000 7f3281505640 1 -- 172.21.15.17:0/597614184 >> v1:172.21.15.94:6789/0 conn(0x7f327c14bba0 legacy=0x7f327c14df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:09.878 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.886+0000 7f3281505640 1 -- 172.21.15.17:0/597614184 >> v1:172.21.15.17:6789/0 conn(0x7f327c0ae250 legacy=0x7f327c142b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:09.878 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.886+0000 7f3281505640 10 monclient: _finish_hunting 0 2024-03-29T11:29:09.878 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.886+0000 7f3281505640 1 monclient: found mon.noname-b 2024-03-29T11:29:09.878 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.886+0000 7f3281505640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:09.878 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.886+0000 7f3281505640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.17:6790/0 2024-03-29T11:29:09.878 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.886+0000 7f3281505640 1 -- 172.21.15.17:0/597614184 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f327c143280 con 0x7f327c0a9720 2024-03-29T11:29:09.879 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.886+0000 7f3281505640 10 monclient: _finish_auth 0 2024-03-29T11:29:09.879 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.886+0000 7f3281505640 10 monclient: _check_auth_tickets 2024-03-29T11:29:09.879 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.886+0000 7f3281505640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:09.879 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.886+0000 7f3281505640 1 -- 172.21.15.17:0/597614184 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f326c0015f0 con 0x7f327c0a9720 2024-03-29T11:29:09.880 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.886+0000 7f3281505640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:09.880 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.886+0000 7f3281505640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:09.880 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.886+0000 7f3281505640 10 monclient: dump: 2024-03-29T11:29:09.880 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:29:09.880 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:29:09.880 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:09.880 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:09.880 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:09.881 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:09.881 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:29:09.881 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:29:09.881 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:29:09.881 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:09.881 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.886+0000 7f328476b640 5 monclient: authenticate success, global_id 4649 2024-03-29T11:29:09.881 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.886+0000 7f328476b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:09.881 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.886+0000 7f3281505640 1 -- 172.21.15.17:0/597614184 <== mon.2 v1:172.21.15.17:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f326c001750 con 0x7f327c0a9720 2024-03-29T11:29:09.881 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.886+0000 7f3281505640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:09.881 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.886+0000 7f328476b640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:09.882 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.886+0000 7f328476b640 4 set_mon_vals no callback set 2024-03-29T11:29:09.884 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.887+0000 7f328476b640 10 monclient: shutdown 2024-03-29T11:29:09.884 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.887+0000 7f328476b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:09.884 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.887+0000 7f328476b640 1 -- 172.21.15.17:0/597614184 >> v1:172.21.15.17:6790/0 conn(0x7f327c0a9720 legacy=0x7f327c13f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:09.884 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.887+0000 7f328476b640 1 -- 172.21.15.17:0/597614184 shutdown_connections 2024-03-29T11:29:09.884 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.887+0000 7f328476b640 1 -- 172.21.15.17:0/597614184 >> 172.21.15.17:0/597614184 conn(0x7f327c0b35e0 msgr2=0x7f327c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:09.884 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.888+0000 7f328476b640 1 -- 172.21.15.17:0/597614184 shutdown_connections 2024-03-29T11:29:09.884 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.888+0000 7f328476b640 1 -- 172.21.15.17:0/597614184 wait complete. 2024-03-29T11:29:09.885 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.889+0000 7f328476b640 10 monclient: build_initial_monmap 2024-03-29T11:29:09.885 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.889+0000 7f328476b640 10 monclient: monmap: 2024-03-29T11:29:09.885 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:29:09.885 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:09.885 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:29:09.885 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:29:09.885 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:09.885 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:09.885 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:29:09.885 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:29:09.885 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:29:09.885 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:09.885 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.889+0000 7f328476b640 1 librados: starting msgr at 2024-03-29T11:29:09.885 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.889+0000 7f328476b640 1 librados: starting objecter 2024-03-29T11:29:09.886 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.889+0000 7f328476b640 1 Processor -- start 2024-03-29T11:29:09.886 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.889+0000 7f328476b640 1 -- start start 2024-03-29T11:29:09.886 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.889+0000 7f328476b640 1 librados: setting wanted keys 2024-03-29T11:29:09.886 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.889+0000 7f328476b640 1 librados: calling monclient init 2024-03-29T11:29:09.886 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.889+0000 7f328476b640 10 monclient: init 2024-03-29T11:29:09.886 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.890+0000 7f328476b640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:09.886 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.890+0000 7f328476b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:29:09.886 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.890+0000 7f328476b640 10 monclient(hunting): picked mon.noname-a con 0x7f327c0a9720 addr v1:172.21.15.17:6789/0 2024-03-29T11:29:09.886 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.890+0000 7f328476b640 10 monclient(hunting): picked mon.noname-b con 0x7f327c0ae250 addr v1:172.21.15.94:6789/0 2024-03-29T11:29:09.886 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.890+0000 7f328476b640 10 monclient(hunting): picked mon.noname-c con 0x7f327c14bba0 addr v1:172.21.15.17:6790/0 2024-03-29T11:29:09.886 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.890+0000 7f328476b640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f327c0726d0 con 0x7f327c0a9720 2024-03-29T11:29:09.886 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.890+0000 7f328476b640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f327c0714d0 con 0x7f327c0ae250 2024-03-29T11:29:09.886 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.890+0000 7f328476b640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f327c0717b0 con 0x7f327c14bba0 2024-03-29T11:29:09.887 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.890+0000 7f328476b640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:09.887 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.890+0000 7f328476b640 10 monclient(hunting): authenticate will time out at 1456.095755s 2024-03-29T11:29:09.893 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.894+0000 7f3282507640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f327c0a9720 0x7f327c071eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:51253/0 (socket says 172.21.15.17:51253) 2024-03-29T11:29:09.893 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.894+0000 7f3282507640 1 -- 172.21.15.17:0/1536657789 learned_addr learned my addr 172.21.15.17:0/1536657789 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:29:09.894 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.896+0000 7f3272ffd640 10 client.?.objecter ms_handle_connect 0x7f327c0a9720 2024-03-29T11:29:09.894 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.896+0000 7f3272ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:09.894 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.896+0000 7f3272ffd640 10 client.?.objecter ms_handle_connect 0x7f327c14bba0 2024-03-29T11:29:09.894 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.896+0000 7f3272ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:09.894 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.896+0000 7f3272ffd640 1 -- 172.21.15.17:0/1536657789 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1017705806 0 0) 0x7f327c0726d0 con 0x7f327c0a9720 2024-03-29T11:29:09.894 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.896+0000 7f3272ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:09.894 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.896+0000 7f3272ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:09.894 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.896+0000 7f3272ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:09.894 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.896+0000 7f3272ffd640 10 monclient(hunting): my global_id is 4554 2024-03-29T11:29:09.894 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.896+0000 7f3272ffd640 1 -- 172.21.15.17:0/1536657789 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3260004eb0 con 0x7f327c0a9720 2024-03-29T11:29:09.894 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.896+0000 7f3272ffd640 1 -- 172.21.15.17:0/1536657789 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 269212633 0 0) 0x7f327c0717b0 con 0x7f327c14bba0 2024-03-29T11:29:09.894 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.896+0000 7f3272ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:09.894 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.896+0000 7f3272ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:09.895 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.897+0000 7f3272ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:09.895 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.897+0000 7f3272ffd640 10 monclient(hunting): my global_id is 4652 2024-03-29T11:29:09.895 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.897+0000 7f3272ffd640 1 -- 172.21.15.17:0/1536657789 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f327c0726d0 con 0x7f327c14bba0 2024-03-29T11:29:09.895 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.897+0000 7f3272ffd640 1 -- 172.21.15.17:0/1536657789 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2180631320 0 0) 0x7f3260004eb0 con 0x7f327c0a9720 2024-03-29T11:29:09.895 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.897+0000 7f3272ffd640 1 -- 172.21.15.17:0/1536657789 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f327c0717b0 con 0x7f327c0a9720 2024-03-29T11:29:09.895 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.897+0000 7f3272ffd640 1 -- 172.21.15.17:0/1536657789 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f3278004080 con 0x7f327c0a9720 2024-03-29T11:29:09.895 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.897+0000 7f3272ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:09.895 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.897+0000 7f3272ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:09.895 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.897+0000 7f3272ffd640 10 monclient(hunting): dump: 2024-03-29T11:29:09.895 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:29:09.895 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:29:09.895 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:09.895 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:09.895 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:09.896 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:09.896 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:29:09.896 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:29:09.896 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:29:09.896 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:09.896 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.897+0000 7f3272ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:09.896 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.898+0000 7f3272ffd640 1 -- 172.21.15.17:0/1536657789 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2173744523 0 0) 0x7f327c0726d0 con 0x7f327c14bba0 2024-03-29T11:29:09.896 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.898+0000 7f3272ffd640 1 -- 172.21.15.17:0/1536657789 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3260004eb0 con 0x7f327c14bba0 2024-03-29T11:29:09.896 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.898+0000 7f3272ffd640 1 -- 172.21.15.17:0/1536657789 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f3274001b90 con 0x7f327c14bba0 2024-03-29T11:29:09.896 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.898+0000 7f3272ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:09.896 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.898+0000 7f3272ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:09.896 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.898+0000 7f3272ffd640 10 monclient(hunting): dump: 2024-03-29T11:29:09.896 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:29:09.896 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:29:09.896 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:09.897 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:09.897 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:09.897 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:09.897 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:29:09.897 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:29:09.897 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:29:09.897 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:09.897 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.898+0000 7f3272ffd640 1 -- 172.21.15.17:0/1536657789 <== mon.0 v1:172.21.15.17:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 661304606 0 0) 0x7f327c0717b0 con 0x7f327c0a9720 2024-03-29T11:29:09.897 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.898+0000 7f3272ffd640 1 -- 172.21.15.17:0/1536657789 >> v1:172.21.15.17:6790/0 conn(0x7f327c14bba0 legacy=0x7f327c070010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:09.897 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.898+0000 7f3272ffd640 1 -- 172.21.15.17:0/1536657789 >> v1:172.21.15.94:6789/0 conn(0x7f327c0ae250 legacy=0x7f327c1e4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:09.897 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.898+0000 7f3272ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:29:09.897 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.898+0000 7f3272ffd640 1 monclient: found mon.a 2024-03-29T11:29:09.897 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.898+0000 7f3272ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:09.898 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.898+0000 7f3272ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:29:09.898 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.898+0000 7f3272ffd640 1 -- 172.21.15.17:0/1536657789 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f327c13e1e0 con 0x7f327c0a9720 2024-03-29T11:29:09.898 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.898+0000 7f3272ffd640 10 monclient: _finish_auth 0 2024-03-29T11:29:09.898 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.898+0000 7f3272ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:29:09.898 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.898+0000 7f3272ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:09.898 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.898+0000 7f328476b640 5 monclient: authenticate success, global_id 4554 2024-03-29T11:29:09.898 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.898+0000 7f328476b640 10 monclient: _renew_subs 2024-03-29T11:29:09.898 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.898+0000 7f328476b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:29:09.898 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.898+0000 7f328476b640 1 -- 172.21.15.17:0/1536657789 --> v1:172.21.15.17:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f327c1e4920 con 0x7f327c0a9720 2024-03-29T11:29:09.898 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.898+0000 7f328476b640 10 client.4554.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:09.898 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.898+0000 7f328476b640 10 monclient: _renew_subs 2024-03-29T11:29:09.898 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.898+0000 7f328476b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:29:09.898 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.898+0000 7f328476b640 1 -- 172.21.15.17:0/1536657789 --> v1:172.21.15.17:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f327c070900 con 0x7f327c0a9720 2024-03-29T11:29:09.898 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.898+0000 7f328476b640 1 librados: init done 2024-03-29T11:29:09.899 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.899+0000 7f3272ffd640 1 -- 172.21.15.17:0/1536657789 <== mon.0 v1:172.21.15.17:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3278000ed0 con 0x7f327c0a9720 2024-03-29T11:29:09.899 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.899+0000 7f3272ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:09.899 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.899+0000 7f3272ffd640 1 -- 172.21.15.17:0/1536657789 <== mon.0 v1:172.21.15.17:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f32780012a0 con 0x7f327c0a9720 2024-03-29T11:29:09.899 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.899+0000 7f3273fff640 4 set_mon_vals no callback set 2024-03-29T11:29:09.899 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.899+0000 7f3272ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:09.899 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.899+0000 7f3272ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:09.899 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.899+0000 7f3272ffd640 10 monclient: dump: 2024-03-29T11:29:09.899 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:29:09.899 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:29:09.899 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:09.899 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:09.899 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:09.899 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:09.899 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:29:09.900 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:29:09.900 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:29:09.900 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:09.900 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.900+0000 7f328476b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:09.900 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.900+0000 7f328476b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:09.900 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.900+0000 7f328476b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:29:09.901 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.900+0000 7f328476b640 1 -- 172.21.15.17:0/1536657789 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3264005080 con 0x7f327c0a9720 2024-03-29T11:29:09.901 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.902+0000 7f3272ffd640 1 -- 172.21.15.17:0/1536657789 <== mon.0 v1:172.21.15.17:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f3278001400 con 0x7f327c0a9720 2024-03-29T11:29:09.901 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.902+0000 7f3272ffd640 1 -- 172.21.15.17:0/1536657789 <== mon.0 v1:172.21.15.17:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3682049282 0 0) 0x7f3278084aa0 con 0x7f327c0a9720 2024-03-29T11:29:09.901 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.902+0000 7f3272ffd640 10 client.4554.objecter ms_dispatch 0x7f327c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:29:09.901 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.902+0000 7f3272ffd640 3 client.4554.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:29:09.901 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.902+0000 7f3272ffd640 3 client.4554.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:29:09.901 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.902+0000 7f3272ffd640 20 client.4554.objecter dump_active .. 0 homeless 2024-03-29T11:29:09.901 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.904+0000 7f3281d06640 10 monclient: get_auth_request con 0x7f326006de90 auth_method 0 2024-03-29T11:29:09.902 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.904+0000 7f3272ffd640 10 client.4554.objecter ms_handle_connect 0x7f326006de90 2024-03-29T11:29:09.903 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.911+0000 7f3272ffd640 1 -- 172.21.15.17:0/1536657789 <== mon.0 v1:172.21.15.17:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3278050380 con 0x7f327c0a9720 2024-03-29T11:29:09.904 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.911+0000 7f3272ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:09.904 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:09.911+0000 7f3272ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:10.238 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:10.248+0000 7f328476b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:10.239 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:10.248+0000 7f328476b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:10.239 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:10.248+0000 7f328476b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:29:10.239 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:10.248+0000 7f328476b640 1 -- 172.21.15.17:0/1536657789 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3264005330 con 0x7f327c0a9720 2024-03-29T11:29:10.242 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:10.251+0000 7f3272ffd640 1 -- 172.21.15.17:0/1536657789 <== mon.0 v1:172.21.15.17:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 543280943) 0x7f32780522b0 con 0x7f327c0a9720 2024-03-29T11:29:10.242 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:10.251+0000 7f3272ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:10.242 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:10.251+0000 7f3272ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:10.242 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:29:10.243 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":11,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:29:01.208095+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:01.208095+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:10.246 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:10.255+0000 7f328476b640 10 librados: watch_flush enter 2024-03-29T11:29:10.246 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:10.255+0000 7f328476b640 10 librados: watch_flush exit 2024-03-29T11:29:10.246 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:10.255+0000 7f328476b640 20 client.4554.objecter shutdown clearing up homeless session... 2024-03-29T11:29:10.246 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:10.255+0000 7f328476b640 10 client.4554.objecter successfully canceled tick 2024-03-29T11:29:10.247 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:10.256+0000 7f328476b640 1 -- 172.21.15.17:0/1536657789 >> v1:172.21.15.17:6816/3236185052 conn(0x7f326006de90 legacy=0x7f3260070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:10.247 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:10.256+0000 7f328476b640 10 monclient: shutdown 2024-03-29T11:29:10.247 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:10.256+0000 7f328476b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:10.247 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:10.256+0000 7f328476b640 1 -- 172.21.15.17:0/1536657789 >> v1:172.21.15.17:6789/0 conn(0x7f327c0a9720 legacy=0x7f327c071eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:10.247 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:10.256+0000 7f328476b640 1 -- 172.21.15.17:0/1536657789 shutdown_connections 2024-03-29T11:29:10.247 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:10.256+0000 7f328476b640 1 -- 172.21.15.17:0/1536657789 >> 172.21.15.17:0/1536657789 conn(0x7f327c0b35e0 msgr2=0x7f327c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:10.248 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:10.257+0000 7f328476b640 1 -- 172.21.15.17:0/1536657789 shutdown_connections 2024-03-29T11:29:10.248 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:10.257+0000 7f328476b640 1 -- 172.21.15.17:0/1536657789 wait complete. 2024-03-29T11:29:10.248 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:10.257+0000 7f328476b640 1 librados: shutdown 2024-03-29T11:29:10.264 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:17.071 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:17.218 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.227+0000 7f02c7886640 10 monclient: get_monmap_and_config 2024-03-29T11:29:17.218 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.227+0000 7f02c7886640 10 monclient: build_initial_monmap 2024-03-29T11:29:17.218 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.227+0000 7f02c7886640 10 monclient: monmap: 2024-03-29T11:29:17.218 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:29:17.219 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:29:17.219 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:29:17.228289+0000 2024-03-29T11:29:17.219 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:29:17.228289+0000 2024-03-29T11:29:17.219 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:17.219 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:17.219 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:29:17.219 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:29:17.219 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:29:17.219 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:17.219 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.229+0000 7f02c7886640 1 Processor -- start 2024-03-29T11:29:17.220 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.229+0000 7f02c7886640 1 -- start start 2024-03-29T11:29:17.220 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.229+0000 7f02c7886640 10 monclient: init 2024-03-29T11:29:17.222 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.230+0000 7f02c7886640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:17.222 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.230+0000 7f02c7886640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:29:17.222 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.230+0000 7f02c7886640 10 monclient(hunting): picked mon.noname-b con 0x7f02c01422b0 addr v1:172.21.15.17:6790/0 2024-03-29T11:29:17.222 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.230+0000 7f02c7886640 10 monclient(hunting): picked mon.noname-a con 0x7f02c0142690 addr v1:172.21.15.17:6789/0 2024-03-29T11:29:17.223 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.230+0000 7f02c7886640 10 monclient(hunting): picked mon.noname-c con 0x7f02c00b17d0 addr v1:172.21.15.94:6789/0 2024-03-29T11:29:17.223 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.230+0000 7f02c7886640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02c0142c80 con 0x7f02c0142690 2024-03-29T11:29:17.223 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.230+0000 7f02c7886640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02c00aae40 con 0x7f02c00b17d0 2024-03-29T11:29:17.223 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.230+0000 7f02c7886640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02c013fbd0 con 0x7f02c01422b0 2024-03-29T11:29:17.223 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.230+0000 7f02c7886640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:17.223 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.230+0000 7f02c7886640 10 monclient(hunting): authenticate will time out at 1463.435551s 2024-03-29T11:29:17.223 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.231+0000 7f02c5622640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f02c01422b0 0x7f02c013f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:56670/0 (socket says 172.21.15.17:56670) 2024-03-29T11:29:17.223 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.231+0000 7f02c5622640 1 -- 172.21.15.17:0/4076108638 learned_addr learned my addr 172.21.15.17:0/4076108638 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:29:17.225 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.234+0000 7f02bffff640 1 -- 172.21.15.17:0/4076108638 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 205125405 0 0) 0x7f02c013fbd0 con 0x7f02c01422b0 2024-03-29T11:29:17.225 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.234+0000 7f02bffff640 10 monclient(hunting): _negotiate 2024-03-29T11:29:17.225 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.234+0000 7f02bffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:17.225 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.234+0000 7f02bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:17.225 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.234+0000 7f02bffff640 10 monclient(hunting): my global_id is 4655 2024-03-29T11:29:17.225 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.234+0000 7f02bffff640 1 -- 172.21.15.17:0/4076108638 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02a8003520 con 0x7f02c01422b0 2024-03-29T11:29:17.228 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.237+0000 7f02bffff640 1 -- 172.21.15.17:0/4076108638 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 378342171 0 0) 0x7f02a8003520 con 0x7f02c01422b0 2024-03-29T11:29:17.228 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.237+0000 7f02bffff640 1 -- 172.21.15.17:0/4076108638 >> v1:172.21.15.94:6789/0 conn(0x7f02c00b17d0 legacy=0x7f02c00b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:17.228 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.237+0000 7f02bffff640 1 -- 172.21.15.17:0/4076108638 >> v1:172.21.15.17:6789/0 conn(0x7f02c0142690 legacy=0x7f02c00b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:17.229 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.237+0000 7f02bffff640 10 monclient: _finish_hunting 0 2024-03-29T11:29:17.229 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.237+0000 7f02bffff640 1 monclient: found mon.noname-b 2024-03-29T11:29:17.229 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.237+0000 7f02bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:17.229 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.237+0000 7f02bffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.17:6790/0 2024-03-29T11:29:17.229 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.237+0000 7f02bffff640 1 -- 172.21.15.17:0/4076108638 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c0151940 con 0x7f02c01422b0 2024-03-29T11:29:17.229 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.237+0000 7f02bffff640 10 monclient: _finish_auth 0 2024-03-29T11:29:17.229 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.237+0000 7f02bffff640 10 monclient: _check_auth_tickets 2024-03-29T11:29:17.229 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.237+0000 7f02bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:17.229 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.237+0000 7f02bffff640 1 -- 172.21.15.17:0/4076108638 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f02b8001690 con 0x7f02c01422b0 2024-03-29T11:29:17.229 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.237+0000 7f02bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:17.229 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.237+0000 7f02bffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:17.229 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.237+0000 7f02bffff640 10 monclient: dump: 2024-03-29T11:29:17.229 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:29:17.229 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:29:17.230 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:17.230 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:17.230 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:17.230 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:17.230 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:29:17.230 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:29:17.230 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:29:17.230 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:17.230 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.237+0000 7f02c7886640 5 monclient: authenticate success, global_id 4655 2024-03-29T11:29:17.230 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.237+0000 7f02c7886640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:17.231 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.238+0000 7f02bffff640 1 -- 172.21.15.17:0/4076108638 <== mon.2 v1:172.21.15.17:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02b80017f0 con 0x7f02c01422b0 2024-03-29T11:29:17.231 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.238+0000 7f02bffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:17.231 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.238+0000 7f02c7886640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:17.231 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.238+0000 7f02c7886640 4 set_mon_vals no callback set 2024-03-29T11:29:17.231 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.238+0000 7f02bffff640 1 -- 172.21.15.17:0/4076108638 <== mon.2 v1:172.21.15.17:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f02b8001c40 con 0x7f02c01422b0 2024-03-29T11:29:17.231 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.238+0000 7f02bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:17.231 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.238+0000 7f02bffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:17.231 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.238+0000 7f02bffff640 10 monclient: dump: 2024-03-29T11:29:17.231 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:29:17.231 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:29:17.231 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:17.232 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:17.232 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:17.232 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:17.232 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:29:17.232 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:29:17.232 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:29:17.232 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:17.232 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.239+0000 7f02c7886640 10 monclient: shutdown 2024-03-29T11:29:17.232 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.239+0000 7f02c7886640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:17.232 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.239+0000 7f02c7886640 1 -- 172.21.15.17:0/4076108638 >> v1:172.21.15.17:6790/0 conn(0x7f02c01422b0 legacy=0x7f02c013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:17.233 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.241+0000 7f02c7886640 1 -- 172.21.15.17:0/4076108638 shutdown_connections 2024-03-29T11:29:17.233 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.242+0000 7f02c7886640 1 -- 172.21.15.17:0/4076108638 >> 172.21.15.17:0/4076108638 conn(0x7f02c0066a00 msgr2=0x7f02c0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:17.233 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.242+0000 7f02c7886640 1 -- 172.21.15.17:0/4076108638 shutdown_connections 2024-03-29T11:29:17.233 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.242+0000 7f02c7886640 1 -- 172.21.15.17:0/4076108638 wait complete. 2024-03-29T11:29:17.234 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.243+0000 7f02c7886640 10 monclient: build_initial_monmap 2024-03-29T11:29:17.234 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.243+0000 7f02c7886640 10 monclient: monmap: 2024-03-29T11:29:17.234 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:29:17.234 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:17.234 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:29:17.234 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:29:17.234 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:17.234 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:17.234 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:29:17.234 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:29:17.234 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:29:17.234 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:17.237 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.244+0000 7f02c7886640 1 librados: starting msgr at 2024-03-29T11:29:17.237 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.244+0000 7f02c7886640 1 librados: starting objecter 2024-03-29T11:29:17.237 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.244+0000 7f02c7886640 1 Processor -- start 2024-03-29T11:29:17.237 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.244+0000 7f02c7886640 1 -- start start 2024-03-29T11:29:17.237 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.244+0000 7f02c7886640 1 librados: setting wanted keys 2024-03-29T11:29:17.237 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.244+0000 7f02c7886640 1 librados: calling monclient init 2024-03-29T11:29:17.237 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.244+0000 7f02c7886640 10 monclient: init 2024-03-29T11:29:17.238 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.245+0000 7f02c7886640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:17.238 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.245+0000 7f02c7886640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:29:17.238 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.245+0000 7f02c7886640 10 monclient(hunting): picked mon.noname-c con 0x7f02c00b17d0 addr v1:172.21.15.17:6790/0 2024-03-29T11:29:17.238 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.246+0000 7f02c7886640 10 monclient(hunting): picked mon.noname-a con 0x7f02c01422b0 addr v1:172.21.15.17:6789/0 2024-03-29T11:29:17.238 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.246+0000 7f02c7886640 10 monclient(hunting): picked mon.noname-b con 0x7f02c0142690 addr v1:172.21.15.94:6789/0 2024-03-29T11:29:17.238 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.246+0000 7f02c7886640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02c00ac8a0 con 0x7f02c01422b0 2024-03-29T11:29:17.238 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.246+0000 7f02c7886640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02c00acb80 con 0x7f02c0142690 2024-03-29T11:29:17.238 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.246+0000 7f02c7886640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02c00ace60 con 0x7f02c00b17d0 2024-03-29T11:29:17.238 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.246+0000 7f02c7886640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:17.238 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.246+0000 7f02c7886640 10 monclient(hunting): authenticate will time out at 1463.451404s 2024-03-29T11:29:17.245 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.249+0000 7f02c4e21640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f02c01422b0 0x7f02c012fb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:36429/0 (socket says 172.21.15.17:36429) 2024-03-29T11:29:17.245 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.249+0000 7f02c4e21640 1 -- 172.21.15.17:0/1493632372 learned_addr learned my addr 172.21.15.17:0/1493632372 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:29:17.245 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.249+0000 7f02bdffb640 10 client.?.objecter ms_handle_connect 0x7f02c01422b0 2024-03-29T11:29:17.245 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.249+0000 7f02bdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:17.246 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.249+0000 7f02bdffb640 10 client.?.objecter ms_handle_connect 0x7f02c00b17d0 2024-03-29T11:29:17.246 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.249+0000 7f02bdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:17.246 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.249+0000 7f02bdffb640 1 -- 172.21.15.17:0/1493632372 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3963549078 0 0) 0x7f02c00ace60 con 0x7f02c00b17d0 2024-03-29T11:29:17.246 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.249+0000 7f02bdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:29:17.246 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.250+0000 7f02bdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:17.246 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.250+0000 7f02bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:17.246 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.250+0000 7f02bdffb640 10 monclient(hunting): my global_id is 4658 2024-03-29T11:29:17.246 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.250+0000 7f02bdffb640 1 -- 172.21.15.17:0/1493632372 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02ac004eb0 con 0x7f02c00b17d0 2024-03-29T11:29:17.246 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.250+0000 7f02bdffb640 1 -- 172.21.15.17:0/1493632372 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2675034752 0 0) 0x7f02c00ac8a0 con 0x7f02c01422b0 2024-03-29T11:29:17.246 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.250+0000 7f02bdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:29:17.246 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.250+0000 7f02bdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:17.246 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.250+0000 7f02bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:17.246 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.250+0000 7f02bdffb640 10 monclient(hunting): my global_id is 4557 2024-03-29T11:29:17.246 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.250+0000 7f02bdffb640 1 -- 172.21.15.17:0/1493632372 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02ac0051d0 con 0x7f02c01422b0 2024-03-29T11:29:17.247 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.250+0000 7f02bdffb640 10 client.?.objecter ms_handle_connect 0x7f02c0142690 2024-03-29T11:29:17.247 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.250+0000 7f02bdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:17.247 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.250+0000 7f02bdffb640 1 -- 172.21.15.17:0/1493632372 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3461557793 0 0) 0x7f02ac004eb0 con 0x7f02c00b17d0 2024-03-29T11:29:17.247 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.250+0000 7f02bdffb640 1 -- 172.21.15.17:0/1493632372 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02c00ac8a0 con 0x7f02c00b17d0 2024-03-29T11:29:17.247 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.250+0000 7f02bdffb640 1 -- 172.21.15.17:0/1493632372 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3373390231 0 0) 0x7f02ac0051d0 con 0x7f02c01422b0 2024-03-29T11:29:17.247 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.251+0000 7f02bdffb640 1 -- 172.21.15.17:0/1493632372 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02ac005a20 con 0x7f02c01422b0 2024-03-29T11:29:17.247 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.251+0000 7f02bdffb640 1 -- 172.21.15.17:0/1493632372 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f02b4001790 con 0x7f02c01422b0 2024-03-29T11:29:17.247 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.251+0000 7f02bdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:17.247 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.251+0000 7f02bdffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:17.247 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.251+0000 7f02bdffb640 10 monclient(hunting): dump: 2024-03-29T11:29:17.247 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:29:17.247 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:29:17.247 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:17.247 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:17.247 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:17.247 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:17.248 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:29:17.248 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:29:17.248 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:29:17.248 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:17.248 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.251+0000 7f02bdffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:17.248 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.251+0000 7f02bdffb640 1 -- 172.21.15.17:0/1493632372 <== mon.0 v1:172.21.15.17:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4076501193 0 0) 0x7f02ac005a20 con 0x7f02c01422b0 2024-03-29T11:29:17.248 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.251+0000 7f02bdffb640 1 -- 172.21.15.17:0/1493632372 >> v1:172.21.15.17:6790/0 conn(0x7f02c00b17d0 legacy=0x7f02c00ab7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:17.248 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.251+0000 7f02bdffb640 1 -- 172.21.15.17:0/1493632372 >> v1:172.21.15.94:6789/0 conn(0x7f02c0142690 legacy=0x7f02c0130290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:17.248 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.251+0000 7f02bdffb640 10 monclient: _finish_hunting 0 2024-03-29T11:29:17.248 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.251+0000 7f02bdffb640 1 monclient: found mon.a 2024-03-29T11:29:17.248 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.251+0000 7f02bdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:17.248 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.251+0000 7f02bdffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:29:17.248 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.251+0000 7f02bdffb640 1 -- 172.21.15.17:0/1493632372 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c0132500 con 0x7f02c01422b0 2024-03-29T11:29:17.248 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.251+0000 7f02bdffb640 10 monclient: _finish_auth 0 2024-03-29T11:29:17.248 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.252+0000 7f02bdffb640 10 monclient: _check_auth_tickets 2024-03-29T11:29:17.249 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.252+0000 7f02bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:17.249 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.252+0000 7f02c7886640 5 monclient: authenticate success, global_id 4557 2024-03-29T11:29:17.249 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.252+0000 7f02c7886640 10 monclient: _renew_subs 2024-03-29T11:29:17.249 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.252+0000 7f02c7886640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:29:17.249 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.252+0000 7f02c7886640 1 -- 172.21.15.17:0/1493632372 --> v1:172.21.15.17:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02c01e2d00 con 0x7f02c01422b0 2024-03-29T11:29:17.249 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.252+0000 7f02c7886640 10 client.4557.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:17.249 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.252+0000 7f02c7886640 10 monclient: _renew_subs 2024-03-29T11:29:17.249 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.252+0000 7f02c7886640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:29:17.249 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.252+0000 7f02c7886640 1 -- 172.21.15.17:0/1493632372 --> v1:172.21.15.17:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f02c01e3070 con 0x7f02c01422b0 2024-03-29T11:29:17.249 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.252+0000 7f02c7886640 1 librados: init done 2024-03-29T11:29:17.249 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.254+0000 7f02c7886640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:17.249 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.254+0000 7f02c7886640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:17.250 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.254+0000 7f02c7886640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:29:17.250 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.254+0000 7f02c7886640 1 -- 172.21.15.17:0/1493632372 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02c013d030 con 0x7f02c01422b0 2024-03-29T11:29:17.251 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.254+0000 7f02bdffb640 1 -- 172.21.15.17:0/1493632372 <== mon.0 v1:172.21.15.17:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02b4001ca0 con 0x7f02c01422b0 2024-03-29T11:29:17.251 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.254+0000 7f02bdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:17.251 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.254+0000 7f02bdffb640 1 -- 172.21.15.17:0/1493632372 <== mon.0 v1:172.21.15.17:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f02b4004a00 con 0x7f02c01422b0 2024-03-29T11:29:17.251 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.254+0000 7f02bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:17.251 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.254+0000 7f02bdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:17.251 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.254+0000 7f02bdffb640 10 monclient: dump: 2024-03-29T11:29:17.251 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:29:17.251 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:29:17.251 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:17.251 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:17.251 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:17.252 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:17.252 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:29:17.252 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:29:17.252 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:29:17.252 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:17.252 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.254+0000 7f02bdffb640 1 -- 172.21.15.17:0/1493632372 <== mon.0 v1:172.21.15.17:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f02b401a9d0 con 0x7f02c01422b0 2024-03-29T11:29:17.252 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.255+0000 7f02bdffb640 1 -- 172.21.15.17:0/1493632372 <== mon.0 v1:172.21.15.17:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3682049282 0 0) 0x7f02b4084a70 con 0x7f02c01422b0 2024-03-29T11:29:17.252 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.255+0000 7f02bdffb640 10 client.4557.objecter ms_dispatch 0x7f02c0000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:29:17.252 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.255+0000 7f02bdffb640 3 client.4557.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:29:17.252 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.255+0000 7f02bdffb640 3 client.4557.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:29:17.252 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.255+0000 7f02bdffb640 20 client.4557.objecter dump_active .. 0 homeless 2024-03-29T11:29:17.252 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.255+0000 7f02c5622640 10 monclient: get_auth_request con 0x7f02ac06e1f0 auth_method 0 2024-03-29T11:29:17.252 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.256+0000 7f02bdffb640 10 client.4557.objecter ms_handle_connect 0x7f02ac06e1f0 2024-03-29T11:29:17.252 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.257+0000 7f02bf7fe640 4 set_mon_vals no callback set 2024-03-29T11:29:17.255 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.264+0000 7f02bdffb640 1 -- 172.21.15.17:0/1493632372 <== mon.0 v1:172.21.15.17:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f02b40503d0 con 0x7f02c01422b0 2024-03-29T11:29:17.255 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.264+0000 7f02bdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:17.255 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.264+0000 7f02bdffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:17.584 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.593+0000 7f02c7886640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:17.584 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.593+0000 7f02c7886640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:17.584 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.593+0000 7f02c7886640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:29:17.584 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.593+0000 7f02c7886640 1 -- 172.21.15.17:0/1493632372 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f02c01329a0 con 0x7f02c01422b0 2024-03-29T11:29:17.584 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.594+0000 7f02bdffb640 1 -- 172.21.15.17:0/1493632372 <== mon.0 v1:172.21.15.17:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 543280943) 0x7f02b4052300 con 0x7f02c01422b0 2024-03-29T11:29:17.585 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.594+0000 7f02bdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:17.585 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.594+0000 7f02bdffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:17.586 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:29:17.586 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":11,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:29:01.208095+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:01.208095+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:17.590 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.597+0000 7f02c7886640 10 librados: watch_flush enter 2024-03-29T11:29:17.591 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.599+0000 7f02c7886640 10 librados: watch_flush exit 2024-03-29T11:29:17.591 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.599+0000 7f02c7886640 20 client.4557.objecter shutdown clearing up homeless session... 2024-03-29T11:29:17.591 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.600+0000 7f02c7886640 10 client.4557.objecter successfully canceled tick 2024-03-29T11:29:17.591 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.600+0000 7f02c7886640 1 -- 172.21.15.17:0/1493632372 >> v1:172.21.15.17:6816/3236185052 conn(0x7f02ac06e1f0 legacy=0x7f02ac0706b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:17.591 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.600+0000 7f02c7886640 10 monclient: shutdown 2024-03-29T11:29:17.591 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.600+0000 7f02c7886640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:17.592 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.600+0000 7f02c7886640 1 -- 172.21.15.17:0/1493632372 >> v1:172.21.15.17:6789/0 conn(0x7f02c01422b0 legacy=0x7f02c012fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:17.592 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.600+0000 7f02c7886640 1 -- 172.21.15.17:0/1493632372 shutdown_connections 2024-03-29T11:29:17.592 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.600+0000 7f02c7886640 1 -- 172.21.15.17:0/1493632372 >> 172.21.15.17:0/1493632372 conn(0x7f02c0066a00 msgr2=0x7f02c00b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:17.592 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.600+0000 7f02c7886640 1 -- 172.21.15.17:0/1493632372 shutdown_connections 2024-03-29T11:29:17.592 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.600+0000 7f02c7886640 1 -- 172.21.15.17:0/1493632372 wait complete. 2024-03-29T11:29:17.592 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:17.600+0000 7f02c7886640 1 librados: shutdown 2024-03-29T11:29:17.608 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:24.415 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:24.568 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.574+0000 7f1c19801640 10 monclient: get_monmap_and_config 2024-03-29T11:29:24.568 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.574+0000 7f1c19801640 10 monclient: build_initial_monmap 2024-03-29T11:29:24.568 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.574+0000 7f1c19801640 10 monclient: monmap: 2024-03-29T11:29:24.568 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:29:24.568 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:29:24.568 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:29:24.576724+0000 2024-03-29T11:29:24.568 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:29:24.576724+0000 2024-03-29T11:29:24.568 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:24.568 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:24.568 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:29:24.569 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:29:24.569 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:29:24.569 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:24.569 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.576+0000 7f1c19801640 1 Processor -- start 2024-03-29T11:29:24.569 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.577+0000 7f1c19801640 1 -- start start 2024-03-29T11:29:24.569 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.577+0000 7f1c19801640 10 monclient: init 2024-03-29T11:29:24.571 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.577+0000 7f1c19801640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:24.571 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.577+0000 7f1c19801640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:29:24.571 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.577+0000 7f1c19801640 10 monclient(hunting): picked mon.noname-b con 0x7f1c0c006410 addr v1:172.21.15.17:6790/0 2024-03-29T11:29:24.571 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.577+0000 7f1c19801640 10 monclient(hunting): picked mon.noname-a con 0x7f1c0c02a640 addr v1:172.21.15.17:6789/0 2024-03-29T11:29:24.571 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.577+0000 7f1c19801640 10 monclient(hunting): picked mon.noname-c con 0x7f1c0c0327d0 addr v1:172.21.15.94:6789/0 2024-03-29T11:29:24.571 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.577+0000 7f1c19801640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c0c02c820 con 0x7f1c0c02a640 2024-03-29T11:29:24.571 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.577+0000 7f1c19801640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c0c02a120 con 0x7f1c0c0327d0 2024-03-29T11:29:24.571 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.577+0000 7f1c19801640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c0c0c75b0 con 0x7f1c0c006410 2024-03-29T11:29:24.571 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.578+0000 7f1c19801640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:24.571 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.578+0000 7f1c19801640 10 monclient(hunting): authenticate will time out at 1470.783889s 2024-03-29T11:29:24.573 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.581+0000 7f1c19000640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f1c0c0327d0 0x7f1c0c0d0350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:50681/0 (socket says 172.21.15.17:50681) 2024-03-29T11:29:24.573 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.581+0000 7f1c19000640 1 -- 172.21.15.17:0/1777021302 learned_addr learned my addr 172.21.15.17:0/1777021302 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:29:24.575 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.582+0000 7f1c12ffd640 1 -- 172.21.15.17:0/1777021302 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3433590617 0 0) 0x7f1c0c0c75b0 con 0x7f1c0c006410 2024-03-29T11:29:24.575 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.582+0000 7f1c12ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:24.575 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.582+0000 7f1c12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:24.575 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.582+0000 7f1c12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:24.575 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.582+0000 7f1c12ffd640 10 monclient(hunting): my global_id is 4661 2024-03-29T11:29:24.575 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.582+0000 7f1c12ffd640 1 -- 172.21.15.17:0/1777021302 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c04003520 con 0x7f1c0c006410 2024-03-29T11:29:24.579 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.584+0000 7f1c12ffd640 1 -- 172.21.15.17:0/1777021302 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1462557863 0 0) 0x7f1c0c02a120 con 0x7f1c0c0327d0 2024-03-29T11:29:24.579 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.584+0000 7f1c12ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:24.580 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.584+0000 7f1c12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:24.580 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.584+0000 7f1c12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:24.580 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.584+0000 7f1c12ffd640 10 monclient(hunting): my global_id is 4606 2024-03-29T11:29:24.580 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.584+0000 7f1c12ffd640 1 -- 172.21.15.17:0/1777021302 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c04003840 con 0x7f1c0c0327d0 2024-03-29T11:29:24.580 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.584+0000 7f1c12ffd640 1 -- 172.21.15.17:0/1777021302 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3277924875 0 0) 0x7f1c04003520 con 0x7f1c0c006410 2024-03-29T11:29:24.580 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.585+0000 7f1c12ffd640 1 -- 172.21.15.17:0/1777021302 >> v1:172.21.15.94:6789/0 conn(0x7f1c0c0327d0 legacy=0x7f1c0c0d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:24.580 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.585+0000 7f1c12ffd640 1 -- 172.21.15.17:0/1777021302 >> v1:172.21.15.17:6789/0 conn(0x7f1c0c02a640 legacy=0x7f1c0c0320c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:24.580 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.585+0000 7f1c12ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:29:24.580 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.585+0000 7f1c12ffd640 1 monclient: found mon.noname-b 2024-03-29T11:29:24.580 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.585+0000 7f1c12ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:24.580 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.585+0000 7f1c12ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.17:6790/0 2024-03-29T11:29:24.580 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.585+0000 7f1c12ffd640 1 -- 172.21.15.17:0/1777021302 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c0c033110 con 0x7f1c0c006410 2024-03-29T11:29:24.580 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.585+0000 7f1c12ffd640 10 monclient: _finish_auth 0 2024-03-29T11:29:24.580 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.585+0000 7f1c12ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:29:24.581 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.585+0000 7f1c12ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:24.581 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.585+0000 7f1c12ffd640 1 -- 172.21.15.17:0/1777021302 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f1c080015f0 con 0x7f1c0c006410 2024-03-29T11:29:24.581 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.585+0000 7f1c12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:24.581 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.585+0000 7f1c12ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:24.581 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.585+0000 7f1c12ffd640 10 monclient: dump: 2024-03-29T11:29:24.581 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:29:24.581 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:29:24.581 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:24.581 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:24.581 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:24.581 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:24.581 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:29:24.581 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:29:24.581 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:29:24.581 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:24.582 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.585+0000 7f1c19801640 5 monclient: authenticate success, global_id 4661 2024-03-29T11:29:24.582 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.585+0000 7f1c19801640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:24.582 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.587+0000 7f1c12ffd640 1 -- 172.21.15.17:0/1777021302 <== mon.2 v1:172.21.15.17:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c080017c0 con 0x7f1c0c006410 2024-03-29T11:29:24.582 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.587+0000 7f1c12ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:24.582 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.587+0000 7f1c19801640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:24.583 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.587+0000 7f1c19801640 4 set_mon_vals no callback set 2024-03-29T11:29:24.583 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.588+0000 7f1c19801640 10 monclient: shutdown 2024-03-29T11:29:24.583 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.588+0000 7f1c19801640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:24.583 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.588+0000 7f1c19801640 1 -- 172.21.15.17:0/1777021302 >> v1:172.21.15.17:6790/0 conn(0x7f1c0c006410 legacy=0x7f1c0c0c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:24.583 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.588+0000 7f1c12ffd640 1 -- 172.21.15.17:0/1777021302 <== mon.2 v1:172.21.15.17:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f1c08001c10 con 0x7f1c0c006410 2024-03-29T11:29:24.583 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.588+0000 7f1c12ffd640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:29:24.583 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.589+0000 7f1c19801640 1 -- 172.21.15.17:0/1777021302 shutdown_connections 2024-03-29T11:29:24.583 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.589+0000 7f1c19801640 1 -- 172.21.15.17:0/1777021302 >> 172.21.15.17:0/1777021302 conn(0x7f1c0c0058f0 msgr2=0x7f1c0c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:24.584 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.591+0000 7f1c19801640 1 -- 172.21.15.17:0/1777021302 shutdown_connections 2024-03-29T11:29:24.584 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.592+0000 7f1c19801640 1 -- 172.21.15.17:0/1777021302 wait complete. 2024-03-29T11:29:24.585 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.593+0000 7f1c19801640 10 monclient: build_initial_monmap 2024-03-29T11:29:24.585 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.593+0000 7f1c19801640 10 monclient: monmap: 2024-03-29T11:29:24.585 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:29:24.585 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:24.585 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:29:24.585 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:29:24.585 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:24.585 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:24.585 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:29:24.586 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:29:24.586 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:29:24.586 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:24.586 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.593+0000 7f1c19801640 1 librados: starting msgr at 2024-03-29T11:29:24.586 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.593+0000 7f1c19801640 1 librados: starting objecter 2024-03-29T11:29:24.587 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.593+0000 7f1c19801640 1 Processor -- start 2024-03-29T11:29:24.589 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.596+0000 7f1c19801640 1 -- start start 2024-03-29T11:29:24.589 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.596+0000 7f1c19801640 1 librados: setting wanted keys 2024-03-29T11:29:24.589 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.596+0000 7f1c19801640 1 librados: calling monclient init 2024-03-29T11:29:24.589 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.596+0000 7f1c19801640 10 monclient: init 2024-03-29T11:29:24.590 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.597+0000 7f1c19801640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:24.590 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.597+0000 7f1c19801640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:29:24.590 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.597+0000 7f1c19801640 10 monclient(hunting): picked mon.noname-c con 0x7f1c0c006410 addr v1:172.21.15.17:6790/0 2024-03-29T11:29:24.590 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.597+0000 7f1c19801640 10 monclient(hunting): picked mon.noname-b con 0x7f1c0c02a640 addr v1:172.21.15.94:6789/0 2024-03-29T11:29:24.591 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.597+0000 7f1c19801640 10 monclient(hunting): picked mon.noname-a con 0x7f1c0c0327d0 addr v1:172.21.15.17:6789/0 2024-03-29T11:29:24.591 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.597+0000 7f1c19801640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c0c15cb80 con 0x7f1c0c0327d0 2024-03-29T11:29:24.591 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.597+0000 7f1c19801640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c0c15bc70 con 0x7f1c0c02a640 2024-03-29T11:29:24.591 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.597+0000 7f1c19801640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c0c15bf50 con 0x7f1c0c006410 2024-03-29T11:29:24.591 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.597+0000 7f1c19801640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:24.591 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.597+0000 7f1c19801640 10 monclient(hunting): authenticate will time out at 1470.803089s 2024-03-29T11:29:24.594 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.600+0000 7f1c13fff640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7f1c0c006410 0x7f1c0c0bfc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:35813/0 (socket says 172.21.15.17:35813) 2024-03-29T11:29:24.594 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.600+0000 7f1c13fff640 1 -- 172.21.15.17:0/1487159564 learned_addr learned my addr 172.21.15.17:0/1487159564 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:29:24.594 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.600+0000 7f1c10ff9640 10 client.?.objecter ms_handle_connect 0x7f1c0c006410 2024-03-29T11:29:24.594 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.600+0000 7f1c10ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:24.596 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.604+0000 7f1c10ff9640 10 client.?.objecter ms_handle_connect 0x7f1c0c0327d0 2024-03-29T11:29:24.596 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.604+0000 7f1c10ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:24.596 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.604+0000 7f1c10ff9640 10 client.?.objecter ms_handle_connect 0x7f1c0c02a640 2024-03-29T11:29:24.596 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.604+0000 7f1c10ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:24.596 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.604+0000 7f1c10ff9640 1 -- 172.21.15.17:0/1487159564 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2642971507 0 0) 0x7f1c0c15bf50 con 0x7f1c0c006410 2024-03-29T11:29:24.596 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.604+0000 7f1c10ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:29:24.596 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.604+0000 7f1c10ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:24.596 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.604+0000 7f1c10ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:24.597 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.604+0000 7f1c10ff9640 10 monclient(hunting): my global_id is 4664 2024-03-29T11:29:24.597 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.604+0000 7f1c10ff9640 1 -- 172.21.15.17:0/1487159564 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bf8004eb0 con 0x7f1c0c006410 2024-03-29T11:29:24.599 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.605+0000 7f1c10ff9640 1 -- 172.21.15.17:0/1487159564 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2690429637 0 0) 0x7f1bf8004eb0 con 0x7f1c0c006410 2024-03-29T11:29:24.599 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.605+0000 7f1c10ff9640 1 -- 172.21.15.17:0/1487159564 --> v1:172.21.15.17:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c0c15bf50 con 0x7f1c0c006410 2024-03-29T11:29:24.599 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.605+0000 7f1c10ff9640 1 -- 172.21.15.17:0/1487159564 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 875369280 0 0) 0x7f1c0c15cb80 con 0x7f1c0c0327d0 2024-03-29T11:29:24.600 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.605+0000 7f1c10ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:29:24.600 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.605+0000 7f1c10ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:24.600 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.605+0000 7f1c10ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:24.600 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.605+0000 7f1c10ff9640 10 monclient(hunting): my global_id is 4560 2024-03-29T11:29:24.600 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.605+0000 7f1c10ff9640 1 -- 172.21.15.17:0/1487159564 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bf8005590 con 0x7f1c0c0327d0 2024-03-29T11:29:24.600 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.605+0000 7f1c10ff9640 1 -- 172.21.15.17:0/1487159564 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f1c080035f0 con 0x7f1c0c006410 2024-03-29T11:29:24.600 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.605+0000 7f1c10ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:24.600 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.605+0000 7f1c10ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:24.600 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.605+0000 7f1c10ff9640 10 monclient(hunting): dump: 2024-03-29T11:29:24.600 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:29:24.600 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:29:24.601 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:24.601 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:24.601 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:24.601 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:24.601 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:29:24.601 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:29:24.601 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:29:24.601 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:24.601 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.605+0000 7f1c10ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:24.601 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.605+0000 7f1c10ff9640 1 -- 172.21.15.17:0/1487159564 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2937070819 0 0) 0x7f1bf8005590 con 0x7f1c0c0327d0 2024-03-29T11:29:24.601 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.606+0000 7f1c10ff9640 1 -- 172.21.15.17:0/1487159564 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c0c15cb80 con 0x7f1c0c0327d0 2024-03-29T11:29:24.601 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.606+0000 7f1c10ff9640 1 -- 172.21.15.17:0/1487159564 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2766879657 0 0) 0x7f1c0c15bc70 con 0x7f1c0c02a640 2024-03-29T11:29:24.601 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.606+0000 7f1c10ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:29:24.601 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.606+0000 7f1c10ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:24.601 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.606+0000 7f1c10ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:24.602 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.606+0000 7f1c10ff9640 10 monclient(hunting): my global_id is 4609 2024-03-29T11:29:24.602 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.606+0000 7f1c10ff9640 1 -- 172.21.15.17:0/1487159564 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bf8005590 con 0x7f1c0c02a640 2024-03-29T11:29:24.602 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.606+0000 7f1c10ff9640 1 -- 172.21.15.17:0/1487159564 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f1c1404fdd0 con 0x7f1c0c0327d0 2024-03-29T11:29:24.602 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.606+0000 7f1c10ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:24.602 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.606+0000 7f1c10ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:24.602 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.606+0000 7f1c10ff9640 10 monclient(hunting): dump: 2024-03-29T11:29:24.602 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:29:24.602 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:29:24.602 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:24.602 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:24.602 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:24.603 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:24.603 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:29:24.603 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:29:24.603 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:29:24.603 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:24.603 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.606+0000 7f1c10ff9640 1 -- 172.21.15.17:0/1487159564 <== mon.2 v1:172.21.15.17:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3813251947 0 0) 0x7f1c0c15bf50 con 0x7f1c0c006410 2024-03-29T11:29:24.603 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.606+0000 7f1c10ff9640 1 -- 172.21.15.17:0/1487159564 >> v1:172.21.15.94:6789/0 conn(0x7f1c0c02a640 legacy=0x7f1c0c15c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:24.603 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.606+0000 7f1c10ff9640 1 -- 172.21.15.17:0/1487159564 >> v1:172.21.15.17:6789/0 conn(0x7f1c0c0327d0 legacy=0x7f1c0c0bae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:24.603 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.606+0000 7f1c10ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:29:24.603 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.606+0000 7f1c10ff9640 1 monclient: found mon.c 2024-03-29T11:29:24.603 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.606+0000 7f1c10ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:24.603 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.606+0000 7f1c10ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:29:24.603 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.606+0000 7f1c10ff9640 1 -- 172.21.15.17:0/1487159564 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c0c0bf4f0 con 0x7f1c0c006410 2024-03-29T11:29:24.604 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.606+0000 7f1c10ff9640 10 monclient: _finish_auth 0 2024-03-29T11:29:24.604 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.606+0000 7f1c10ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:29:24.604 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.606+0000 7f1c10ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:24.604 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.606+0000 7f1c19801640 5 monclient: authenticate success, global_id 4664 2024-03-29T11:29:24.604 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.606+0000 7f1c19801640 10 monclient: _renew_subs 2024-03-29T11:29:24.604 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.606+0000 7f1c19801640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:29:24.604 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.606+0000 7f1c19801640 1 -- 172.21.15.17:0/1487159564 --> v1:172.21.15.17:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c0c0bb570 con 0x7f1c0c006410 2024-03-29T11:29:24.607 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.606+0000 7f1c19801640 10 client.4664.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:24.607 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.606+0000 7f1c19801640 10 monclient: _renew_subs 2024-03-29T11:29:24.607 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.606+0000 7f1c19801640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:29:24.607 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.606+0000 7f1c19801640 1 -- 172.21.15.17:0/1487159564 --> v1:172.21.15.17:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c0c0bf100 con 0x7f1c0c006410 2024-03-29T11:29:24.607 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.606+0000 7f1c19801640 1 librados: init done 2024-03-29T11:29:24.607 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.609+0000 7f1c10ff9640 1 -- 172.21.15.17:0/1487159564 <== mon.2 v1:172.21.15.17:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c08003ac0 con 0x7f1c0c006410 2024-03-29T11:29:24.607 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.609+0000 7f1c10ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:24.607 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.609+0000 7f1c10ff9640 1 -- 172.21.15.17:0/1487159564 <== mon.2 v1:172.21.15.17:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f1c080035f0 con 0x7f1c0c006410 2024-03-29T11:29:24.607 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.609+0000 7f1c10ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:24.607 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.609+0000 7f1c10ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:24.607 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.609+0000 7f1c10ff9640 10 monclient: dump: 2024-03-29T11:29:24.608 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:29:24.608 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:29:24.608 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:24.608 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:24.608 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:24.608 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:24.608 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:29:24.608 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:29:24.608 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:29:24.608 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:24.608 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.609+0000 7f1c11ffb640 4 set_mon_vals no callback set 2024-03-29T11:29:24.608 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.610+0000 7f1c19801640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:24.608 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.610+0000 7f1c19801640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:24.609 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.610+0000 7f1c19801640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:29:24.609 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.610+0000 7f1c19801640 1 -- 172.21.15.17:0/1487159564 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bf4005080 con 0x7f1c0c006410 2024-03-29T11:29:24.609 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.611+0000 7f1c10ff9640 1 -- 172.21.15.17:0/1487159564 <== mon.2 v1:172.21.15.17:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f1c08003d30 con 0x7f1c0c006410 2024-03-29T11:29:24.609 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.612+0000 7f1c10ff9640 1 -- 172.21.15.17:0/1487159564 <== mon.2 v1:172.21.15.17:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3682049282 0 0) 0x7f1c08084880 con 0x7f1c0c006410 2024-03-29T11:29:24.609 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.612+0000 7f1c10ff9640 10 client.4664.objecter ms_dispatch 0x7f1c14000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:29:24.609 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.612+0000 7f1c10ff9640 3 client.4664.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:29:24.609 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.612+0000 7f1c10ff9640 3 client.4664.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:29:24.609 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.612+0000 7f1c10ff9640 20 client.4664.objecter dump_active .. 0 homeless 2024-03-29T11:29:24.609 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.614+0000 7f1c137fe640 10 monclient: get_auth_request con 0x7f1bf806e2f0 auth_method 0 2024-03-29T11:29:24.609 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.615+0000 7f1c10ff9640 10 client.4664.objecter ms_handle_connect 0x7f1bf806e2f0 2024-03-29T11:29:24.615 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.619+0000 7f1c10ff9640 1 -- 172.21.15.17:0/1487159564 <== mon.2 v1:172.21.15.17:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1c08050160 con 0x7f1c0c006410 2024-03-29T11:29:24.615 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.619+0000 7f1c10ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:24.615 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.619+0000 7f1c10ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:24.940 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.947+0000 7f1c19801640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:24.940 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.947+0000 7f1c19801640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:24.940 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.947+0000 7f1c19801640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.17:6790/0 2024-03-29T11:29:24.940 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.947+0000 7f1c19801640 1 -- 172.21.15.17:0/1487159564 --> v1:172.21.15.17:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1bf4005330 con 0x7f1c0c006410 2024-03-29T11:29:24.942 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.950+0000 7f1c10ff9640 1 -- 172.21.15.17:0/1487159564 <== mon.2 v1:172.21.15.17:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 543280943) 0x7f1c08050160 con 0x7f1c0c006410 2024-03-29T11:29:24.942 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.950+0000 7f1c10ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:24.942 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.950+0000 7f1c10ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:24.942 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:29:24.942 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":11,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:29:01.208095+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:01.208095+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:24.945 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.953+0000 7f1c19801640 10 librados: watch_flush enter 2024-03-29T11:29:24.948 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.955+0000 7f1c19801640 10 librados: watch_flush exit 2024-03-29T11:29:24.949 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.955+0000 7f1c19801640 20 client.4664.objecter shutdown clearing up homeless session... 2024-03-29T11:29:24.949 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.955+0000 7f1c19801640 10 client.4664.objecter successfully canceled tick 2024-03-29T11:29:24.949 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.957+0000 7f1c19801640 1 -- 172.21.15.17:0/1487159564 >> v1:172.21.15.17:6816/3236185052 conn(0x7f1bf806e2f0 legacy=0x7f1bf80707b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:24.949 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.957+0000 7f1c19801640 10 monclient: shutdown 2024-03-29T11:29:24.949 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.957+0000 7f1c19801640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:24.950 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.957+0000 7f1c19801640 1 -- 172.21.15.17:0/1487159564 >> v1:172.21.15.17:6790/0 conn(0x7f1c0c006410 legacy=0x7f1c0c0bfc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:24.952 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.958+0000 7f1c19801640 1 -- 172.21.15.17:0/1487159564 shutdown_connections 2024-03-29T11:29:24.952 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.958+0000 7f1c19801640 1 -- 172.21.15.17:0/1487159564 >> 172.21.15.17:0/1487159564 conn(0x7f1c0c0058f0 msgr2=0x7f1c0c037ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:24.953 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.960+0000 7f1c19801640 1 -- 172.21.15.17:0/1487159564 shutdown_connections 2024-03-29T11:29:24.953 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.961+0000 7f1c19801640 1 -- 172.21.15.17:0/1487159564 wait complete. 2024-03-29T11:29:24.954 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:24.962+0000 7f1c19801640 1 librados: shutdown 2024-03-29T11:29:24.972 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:31.780 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:31.942 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.950+0000 7fe4d6a89640 10 monclient: get_monmap_and_config 2024-03-29T11:29:31.942 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.950+0000 7fe4d6a89640 10 monclient: build_initial_monmap 2024-03-29T11:29:31.942 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.950+0000 7fe4d6a89640 10 monclient: monmap: 2024-03-29T11:29:31.942 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:29:31.942 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:29:31.942 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:29:31.951250+0000 2024-03-29T11:29:31.942 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:29:31.951250+0000 2024-03-29T11:29:31.942 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:31.943 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:31.943 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:29:31.943 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:29:31.943 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:29:31.943 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:31.946 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.951+0000 7fe4d6a89640 1 Processor -- start 2024-03-29T11:29:31.946 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.951+0000 7fe4d6a89640 1 -- start start 2024-03-29T11:29:31.946 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.951+0000 7fe4d6a89640 10 monclient: init 2024-03-29T11:29:31.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.952+0000 7fe4d6a89640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:31.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.952+0000 7fe4d6a89640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:29:31.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.952+0000 7fe4d6a89640 10 monclient(hunting): picked mon.noname-c con 0x7fe4d01457f0 addr v1:172.21.15.94:6789/0 2024-03-29T11:29:31.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.952+0000 7fe4d6a89640 10 monclient(hunting): picked mon.noname-b con 0x7fe4d013f650 addr v1:172.21.15.17:6790/0 2024-03-29T11:29:31.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.952+0000 7fe4d6a89640 10 monclient(hunting): picked mon.noname-a con 0x7fe4d00a9d70 addr v1:172.21.15.17:6789/0 2024-03-29T11:29:31.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.952+0000 7fe4d6a89640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4d00aaef0 con 0x7fe4d00a9d70 2024-03-29T11:29:31.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.952+0000 7fe4d6a89640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4d0145bd0 con 0x7fe4d01457f0 2024-03-29T11:29:31.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.952+0000 7fe4d6a89640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4d0145eb0 con 0x7fe4d013f650 2024-03-29T11:29:31.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.952+0000 7fe4d6a89640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:31.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.952+0000 7fe4d6a89640 10 monclient(hunting): authenticate will time out at 1478.157278s 2024-03-29T11:29:31.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.955+0000 7fe4cffff640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7fe4d013f650 0x7fe4d00b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:56567/0 (socket says 172.21.15.17:56567) 2024-03-29T11:29:31.947 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.955+0000 7fe4cffff640 1 -- 172.21.15.17:0/2929600828 learned_addr learned my addr 172.21.15.17:0/2929600828 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:29:31.948 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.956+0000 7fe4cf7fe640 1 -- 172.21.15.17:0/2929600828 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 99691858 0 0) 0x7fe4d0145bd0 con 0x7fe4d01457f0 2024-03-29T11:29:31.948 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.956+0000 7fe4cf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:31.948 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.956+0000 7fe4cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:31.948 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.956+0000 7fe4cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:31.948 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.956+0000 7fe4cf7fe640 10 monclient(hunting): my global_id is 4612 2024-03-29T11:29:31.948 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.956+0000 7fe4cf7fe640 1 -- 172.21.15.17:0/2929600828 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4b8003520 con 0x7fe4d01457f0 2024-03-29T11:29:31.949 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.956+0000 7fe4cf7fe640 1 -- 172.21.15.17:0/2929600828 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2263480012 0 0) 0x7fe4d0145eb0 con 0x7fe4d013f650 2024-03-29T11:29:31.949 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.956+0000 7fe4cf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:31.949 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.956+0000 7fe4cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:31.949 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.956+0000 7fe4cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:31.950 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.956+0000 7fe4cf7fe640 10 monclient(hunting): my global_id is 4667 2024-03-29T11:29:31.950 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.956+0000 7fe4cf7fe640 1 -- 172.21.15.17:0/2929600828 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4d0145bd0 con 0x7fe4d013f650 2024-03-29T11:29:31.950 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.956+0000 7fe4cf7fe640 1 -- 172.21.15.17:0/2929600828 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1723430894 0 0) 0x7fe4b8003520 con 0x7fe4d01457f0 2024-03-29T11:29:31.950 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.956+0000 7fe4cf7fe640 1 -- 172.21.15.17:0/2929600828 >> v1:172.21.15.17:6790/0 conn(0x7fe4d013f650 legacy=0x7fe4d00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:31.951 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.956+0000 7fe4cf7fe640 1 -- 172.21.15.17:0/2929600828 >> v1:172.21.15.17:6789/0 conn(0x7fe4d00a9d70 legacy=0x7fe4d014d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:31.954 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.958+0000 7fe4cf7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:29:31.955 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.958+0000 7fe4cf7fe640 1 monclient: found mon.noname-c 2024-03-29T11:29:31.955 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.958+0000 7fe4cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:31.955 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.958+0000 7fe4cf7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6789/0 2024-03-29T11:29:31.955 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.958+0000 7fe4cf7fe640 1 -- 172.21.15.17:0/2929600828 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4d0146340 con 0x7fe4d01457f0 2024-03-29T11:29:31.955 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.958+0000 7fe4cf7fe640 10 monclient: _finish_auth 0 2024-03-29T11:29:31.955 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.958+0000 7fe4cf7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:29:31.955 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.958+0000 7fe4cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:31.955 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.958+0000 7fe4d6a89640 5 monclient: authenticate success, global_id 4612 2024-03-29T11:29:31.956 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.958+0000 7fe4d6a89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:31.956 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.958+0000 7fe4cf7fe640 1 -- 172.21.15.17:0/2929600828 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fe4c0001690 con 0x7fe4d01457f0 2024-03-29T11:29:31.956 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.958+0000 7fe4cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:31.956 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.958+0000 7fe4cf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:31.956 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.958+0000 7fe4cf7fe640 10 monclient: dump: 2024-03-29T11:29:31.956 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:29:31.956 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:29:31.956 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:31.956 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:31.956 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:31.956 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:31.956 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:29:31.956 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:29:31.956 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:29:31.957 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:31.957 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.958+0000 7fe4d6a89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:31.957 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.958+0000 7fe4cf7fe640 1 -- 172.21.15.17:0/2929600828 <== mon.1 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4c00018b0 con 0x7fe4d01457f0 2024-03-29T11:29:31.957 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.958+0000 7fe4cf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:31.957 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.958+0000 7fe4d6a89640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:31.957 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.958+0000 7fe4d6a89640 4 set_mon_vals no callback set 2024-03-29T11:29:31.957 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.959+0000 7fe4d6a89640 10 monclient: shutdown 2024-03-29T11:29:31.957 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.959+0000 7fe4d6a89640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:31.957 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.959+0000 7fe4d6a89640 1 -- 172.21.15.17:0/2929600828 >> v1:172.21.15.94:6789/0 conn(0x7fe4d01457f0 legacy=0x7fe4d013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:31.957 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.959+0000 7fe4d6a89640 1 -- 172.21.15.17:0/2929600828 shutdown_connections 2024-03-29T11:29:31.957 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.959+0000 7fe4d6a89640 1 -- 172.21.15.17:0/2929600828 >> 172.21.15.17:0/2929600828 conn(0x7fe4d0066a00 msgr2=0x7fe4d0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:31.957 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.960+0000 7fe4d6a89640 1 -- 172.21.15.17:0/2929600828 shutdown_connections 2024-03-29T11:29:31.958 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.960+0000 7fe4d6a89640 1 -- 172.21.15.17:0/2929600828 wait complete. 2024-03-29T11:29:31.958 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.961+0000 7fe4d6a89640 10 monclient: build_initial_monmap 2024-03-29T11:29:31.958 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.961+0000 7fe4d6a89640 10 monclient: monmap: 2024-03-29T11:29:31.958 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:29:31.958 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:31.958 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:29:31.958 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:29:31.958 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:31.958 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:31.958 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:29:31.958 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:29:31.958 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:29:31.959 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:31.959 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.961+0000 7fe4d6a89640 1 librados: starting msgr at 2024-03-29T11:29:31.959 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.961+0000 7fe4d6a89640 1 librados: starting objecter 2024-03-29T11:29:31.959 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.962+0000 7fe4d6a89640 1 Processor -- start 2024-03-29T11:29:31.969 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.963+0000 7fe4d6a89640 1 -- start start 2024-03-29T11:29:31.969 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.963+0000 7fe4d6a89640 1 librados: setting wanted keys 2024-03-29T11:29:31.969 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.963+0000 7fe4d6a89640 1 librados: calling monclient init 2024-03-29T11:29:31.969 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.963+0000 7fe4d6a89640 10 monclient: init 2024-03-29T11:29:31.969 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.964+0000 7fe4d6a89640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:31.969 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.964+0000 7fe4d6a89640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:29:31.969 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.964+0000 7fe4d6a89640 10 monclient(hunting): picked mon.noname-c con 0x7fe4d00a9d70 addr v1:172.21.15.17:6790/0 2024-03-29T11:29:31.969 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.964+0000 7fe4d6a89640 10 monclient(hunting): picked mon.noname-b con 0x7fe4d013f650 addr v1:172.21.15.94:6789/0 2024-03-29T11:29:31.970 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.964+0000 7fe4d6a89640 10 monclient(hunting): picked mon.noname-a con 0x7fe4d01457f0 addr v1:172.21.15.17:6789/0 2024-03-29T11:29:31.970 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.964+0000 7fe4d6a89640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4d00abcf0 con 0x7fe4d01457f0 2024-03-29T11:29:31.970 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.964+0000 7fe4d6a89640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4d013b170 con 0x7fe4d013f650 2024-03-29T11:29:31.971 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.964+0000 7fe4d6a89640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4d00abf60 con 0x7fe4d00a9d70 2024-03-29T11:29:31.971 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.964+0000 7fe4d6a89640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:31.971 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.964+0000 7fe4d6a89640 10 monclient(hunting): authenticate will time out at 1478.169820s 2024-03-29T11:29:31.971 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.968+0000 7fe4cffff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fe4d013f650 0x7fe4d013a350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:49784/0 (socket says 172.21.15.17:49784) 2024-03-29T11:29:31.971 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.968+0000 7fe4cffff640 1 -- 172.21.15.17:0/546618063 learned_addr learned my addr 172.21.15.17:0/546618063 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:29:31.971 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.968+0000 7fe4d4825640 1 --1- >> v1:172.21.15.17:6790/0 conn(0x7fe4d00a9d70 0x7fe4d006de90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6790/0 says I am v1:172.21.15.17:56619/0 (socket says 172.21.15.17:56619) 2024-03-29T11:29:31.971 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.968+0000 7fe4cd7fa640 10 client.?.objecter ms_handle_connect 0x7fe4d013f650 2024-03-29T11:29:31.971 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.968+0000 7fe4cd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:31.972 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.968+0000 7fe4cd7fa640 10 client.?.objecter ms_handle_connect 0x7fe4d00a9d70 2024-03-29T11:29:31.972 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.968+0000 7fe4cd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:31.972 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.969+0000 7fe4cd7fa640 1 -- 172.21.15.17:0/546618063 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3036656486 0 0) 0x7fe4d00abf60 con 0x7fe4d00a9d70 2024-03-29T11:29:31.972 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.969+0000 7fe4cd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:29:31.972 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.969+0000 7fe4cd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:31.972 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.969+0000 7fe4cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:31.972 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.969+0000 7fe4cd7fa640 10 monclient(hunting): my global_id is 4670 2024-03-29T11:29:31.972 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.969+0000 7fe4cd7fa640 1 -- 172.21.15.17:0/546618063 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4bc004eb0 con 0x7fe4d00a9d70 2024-03-29T11:29:31.972 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.969+0000 7fe4cd7fa640 1 -- 172.21.15.17:0/546618063 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 531706428 0 0) 0x7fe4d013b170 con 0x7fe4d013f650 2024-03-29T11:29:31.972 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.969+0000 7fe4cd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:29:31.972 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.969+0000 7fe4cd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:31.972 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.969+0000 7fe4cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:31.972 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.969+0000 7fe4cd7fa640 10 monclient(hunting): my global_id is 4615 2024-03-29T11:29:31.973 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.969+0000 7fe4cd7fa640 1 -- 172.21.15.17:0/546618063 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4d00abf60 con 0x7fe4d013f650 2024-03-29T11:29:31.973 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.970+0000 7fe4cd7fa640 1 -- 172.21.15.17:0/546618063 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 273453122 0 0) 0x7fe4d00abf60 con 0x7fe4d013f650 2024-03-29T11:29:31.973 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.970+0000 7fe4cd7fa640 1 -- 172.21.15.17:0/546618063 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4d013b170 con 0x7fe4d013f650 2024-03-29T11:29:31.973 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.970+0000 7fe4cd7fa640 1 -- 172.21.15.17:0/546618063 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fe4c8001cb0 con 0x7fe4d013f650 2024-03-29T11:29:31.973 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.970+0000 7fe4cd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:31.973 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.970+0000 7fe4cd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:31.973 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.970+0000 7fe4cd7fa640 10 monclient(hunting): dump: 2024-03-29T11:29:31.973 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:29:31.973 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:29:31.973 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:31.973 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:31.973 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:31.973 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:31.973 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:29:31.974 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:29:31.974 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:29:31.974 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:31.974 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.970+0000 7fe4cd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:31.974 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.970+0000 7fe4cd7fa640 1 -- 172.21.15.17:0/546618063 <== mon.1 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3155250137 0 0) 0x7fe4d013b170 con 0x7fe4d013f650 2024-03-29T11:29:31.974 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.970+0000 7fe4cd7fa640 1 -- 172.21.15.17:0/546618063 >> v1:172.21.15.17:6790/0 conn(0x7fe4d00a9d70 legacy=0x7fe4d006de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:31.974 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.970+0000 7fe4cd7fa640 1 -- 172.21.15.17:0/546618063 >> v1:172.21.15.17:6789/0 conn(0x7fe4d01457f0 legacy=0x7fe4d013aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:31.974 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.970+0000 7fe4cd7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:29:31.974 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.970+0000 7fe4cd7fa640 1 monclient: found mon.b 2024-03-29T11:29:31.974 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.970+0000 7fe4cd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:31.974 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.970+0000 7fe4cd7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:29:31.974 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.970+0000 7fe4cd7fa640 1 -- 172.21.15.17:0/546618063 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4d00ac240 con 0x7fe4d013f650 2024-03-29T11:29:31.974 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.970+0000 7fe4cd7fa640 10 monclient: _finish_auth 0 2024-03-29T11:29:31.974 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.970+0000 7fe4cd7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:29:31.974 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.970+0000 7fe4cd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:31.975 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.970+0000 7fe4d6a89640 5 monclient: authenticate success, global_id 4615 2024-03-29T11:29:31.975 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.970+0000 7fe4d6a89640 10 monclient: _renew_subs 2024-03-29T11:29:31.975 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.970+0000 7fe4d6a89640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:29:31.975 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.970+0000 7fe4d6a89640 1 -- 172.21.15.17:0/546618063 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4d0071c40 con 0x7fe4d013f650 2024-03-29T11:29:31.975 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.971+0000 7fe4d6a89640 10 client.4615.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:31.975 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.971+0000 7fe4d6a89640 10 monclient: _renew_subs 2024-03-29T11:29:31.975 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.971+0000 7fe4d6a89640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:29:31.975 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.971+0000 7fe4d6a89640 1 -- 172.21.15.17:0/546618063 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4d006e780 con 0x7fe4d013f650 2024-03-29T11:29:31.975 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.971+0000 7fe4d6a89640 1 librados: init done 2024-03-29T11:29:31.975 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.971+0000 7fe4cd7fa640 1 -- 172.21.15.17:0/546618063 <== mon.1 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4c8004b60 con 0x7fe4d013f650 2024-03-29T11:29:31.975 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.971+0000 7fe4cd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:31.975 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.971+0000 7fe4cd7fa640 1 -- 172.21.15.17:0/546618063 <== mon.1 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7fe4c8004e70 con 0x7fe4d013f650 2024-03-29T11:29:31.975 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.971+0000 7fe4cd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:31.975 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.971+0000 7fe4cd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:31.975 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.971+0000 7fe4cd7fa640 10 monclient: dump: 2024-03-29T11:29:31.976 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:29:31.976 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:29:31.976 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:31.976 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:31.976 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:31.976 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:31.976 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:29:31.976 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:29:31.976 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:29:31.976 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:31.976 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.971+0000 7fe4ceffd640 4 set_mon_vals no callback set 2024-03-29T11:29:31.976 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.972+0000 7fe4d6a89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:31.976 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.973+0000 7fe4d6a89640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:31.976 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.973+0000 7fe4d6a89640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:29:31.977 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.973+0000 7fe4d6a89640 1 -- 172.21.15.17:0/546618063 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4b0005080 con 0x7fe4d013f650 2024-03-29T11:29:31.977 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.974+0000 7fe4cd7fa640 1 -- 172.21.15.17:0/546618063 <== mon.1 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7fe4c801ae40 con 0x7fe4d013f650 2024-03-29T11:29:31.977 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.975+0000 7fe4d4825640 10 monclient: get_auth_request con 0x7fe4bc06dec0 auth_method 0 2024-03-29T11:29:31.977 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.975+0000 7fe4cd7fa640 10 client.4615.objecter ms_handle_connect 0x7fe4bc06dec0 2024-03-29T11:29:31.977 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.975+0000 7fe4cd7fa640 1 -- 172.21.15.17:0/546618063 <== mon.1 v1:172.21.15.94:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3682049282 0 0) 0x7fe4c8084df0 con 0x7fe4d013f650 2024-03-29T11:29:31.977 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.975+0000 7fe4cd7fa640 10 client.4615.objecter ms_dispatch 0x7fe4d0000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:29:31.977 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.975+0000 7fe4cd7fa640 3 client.4615.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:29:31.977 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.975+0000 7fe4cd7fa640 3 client.4615.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:29:31.977 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.975+0000 7fe4cd7fa640 20 client.4615.objecter dump_active .. 0 homeless 2024-03-29T11:29:31.977 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.980+0000 7fe4cd7fa640 1 -- 172.21.15.17:0/546618063 <== mon.1 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe4c8050500 con 0x7fe4d013f650 2024-03-29T11:29:31.978 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.980+0000 7fe4cd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:31.978 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:31.980+0000 7fe4cd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:32.336 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:32.344+0000 7fe4d6a89640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:32.336 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:32.344+0000 7fe4d6a89640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:32.336 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:32.344+0000 7fe4d6a89640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:29:32.336 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:32.344+0000 7fe4d6a89640 1 -- 172.21.15.17:0/546618063 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4b0005330 con 0x7fe4d013f650 2024-03-29T11:29:32.337 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:32.345+0000 7fe4cd7fa640 1 -- 172.21.15.17:0/546618063 <== mon.1 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 543280943) 0x7fe4c8052600 con 0x7fe4d013f650 2024-03-29T11:29:32.337 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:32.345+0000 7fe4cd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:32.337 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:32.345+0000 7fe4cd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:32.337 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:29:32.337 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":11,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:29:01.208095+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:01.208095+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:32.340 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:32.349+0000 7fe4d6a89640 10 librados: watch_flush enter 2024-03-29T11:29:32.340 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:32.349+0000 7fe4d6a89640 10 librados: watch_flush exit 2024-03-29T11:29:32.340 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:32.349+0000 7fe4d6a89640 20 client.4615.objecter shutdown clearing up homeless session... 2024-03-29T11:29:32.340 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:32.349+0000 7fe4d6a89640 10 client.4615.objecter successfully canceled tick 2024-03-29T11:29:32.341 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:32.349+0000 7fe4d6a89640 1 -- 172.21.15.17:0/546618063 >> v1:172.21.15.17:6816/3236185052 conn(0x7fe4bc06dec0 legacy=0x7fe4bc070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:32.341 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:32.349+0000 7fe4d6a89640 10 monclient: shutdown 2024-03-29T11:29:32.341 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:32.349+0000 7fe4d6a89640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:32.341 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:32.349+0000 7fe4d6a89640 1 -- 172.21.15.17:0/546618063 >> v1:172.21.15.94:6789/0 conn(0x7fe4d013f650 legacy=0x7fe4d013a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:32.341 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:32.350+0000 7fe4d6a89640 1 -- 172.21.15.17:0/546618063 shutdown_connections 2024-03-29T11:29:32.341 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:32.350+0000 7fe4d6a89640 1 -- 172.21.15.17:0/546618063 >> 172.21.15.17:0/546618063 conn(0x7fe4d0066a00 msgr2=0x7fe4d0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:32.342 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:32.350+0000 7fe4d6a89640 1 -- 172.21.15.17:0/546618063 shutdown_connections 2024-03-29T11:29:32.342 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:32.350+0000 7fe4d6a89640 1 -- 172.21.15.17:0/546618063 wait complete. 2024-03-29T11:29:32.342 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:32.350+0000 7fe4d6a89640 1 librados: shutdown 2024-03-29T11:29:32.360 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:39.167 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:39.317 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.325+0000 7f8b7273d640 10 monclient: get_monmap_and_config 2024-03-29T11:29:39.317 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.325+0000 7f8b7273d640 10 monclient: build_initial_monmap 2024-03-29T11:29:39.317 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.325+0000 7f8b7273d640 10 monclient: monmap: 2024-03-29T11:29:39.317 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:29:39.317 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:29:39.318 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:29:39.326477+0000 2024-03-29T11:29:39.318 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:29:39.326477+0000 2024-03-29T11:29:39.318 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:39.318 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:39.318 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:29:39.318 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:29:39.318 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:29:39.318 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:39.320 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.326+0000 7f8b7273d640 1 Processor -- start 2024-03-29T11:29:39.320 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.326+0000 7f8b7273d640 1 -- start start 2024-03-29T11:29:39.320 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.326+0000 7f8b7273d640 10 monclient: init 2024-03-29T11:29:39.320 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.327+0000 7f8b7273d640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:39.320 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.327+0000 7f8b7273d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:29:39.320 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.327+0000 7f8b7273d640 10 monclient(hunting): picked mon.noname-a con 0x7f8b6c0b0770 addr v1:172.21.15.17:6789/0 2024-03-29T11:29:39.320 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.327+0000 7f8b7273d640 10 monclient(hunting): picked mon.noname-b con 0x7f8b6c14a700 addr v1:172.21.15.17:6790/0 2024-03-29T11:29:39.320 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.327+0000 7f8b7273d640 10 monclient(hunting): picked mon.noname-c con 0x7f8b6c0ac5a0 addr v1:172.21.15.94:6789/0 2024-03-29T11:29:39.321 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.327+0000 7f8b7273d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b6c0aba70 con 0x7f8b6c0b0770 2024-03-29T11:29:39.321 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.328+0000 7f8b7273d640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b6c0aecb0 con 0x7f8b6c0ac5a0 2024-03-29T11:29:39.321 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.328+0000 7f8b7273d640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b6c0af150 con 0x7f8b6c14a700 2024-03-29T11:29:39.321 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.328+0000 7f8b7273d640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:39.321 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.328+0000 7f8b7273d640 10 monclient(hunting): authenticate will time out at 1485.533092s 2024-03-29T11:29:39.321 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.328+0000 7f8b70cda640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f8b6c0ac5a0 0x7f8b6c151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:38378/0 (socket says 172.21.15.17:38378) 2024-03-29T11:29:39.321 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.328+0000 7f8b70cda640 1 -- 172.21.15.17:0/3242905765 learned_addr learned my addr 172.21.15.17:0/3242905765 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:29:39.331 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.329+0000 7f8b6affd640 1 -- 172.21.15.17:0/3242905765 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1802432649 0 0) 0x7f8b6c0aba70 con 0x7f8b6c0b0770 2024-03-29T11:29:39.331 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.329+0000 7f8b6affd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:39.331 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.329+0000 7f8b6affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:39.332 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.329+0000 7f8b6affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:39.332 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.329+0000 7f8b6affd640 10 monclient(hunting): my global_id is 4563 2024-03-29T11:29:39.332 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.329+0000 7f8b6affd640 1 -- 172.21.15.17:0/3242905765 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b58003520 con 0x7f8b6c0b0770 2024-03-29T11:29:39.332 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.329+0000 7f8b6affd640 1 -- 172.21.15.17:0/3242905765 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3821343266 0 0) 0x7f8b6c0aecb0 con 0x7f8b6c0ac5a0 2024-03-29T11:29:39.332 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.329+0000 7f8b6affd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:39.332 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.329+0000 7f8b6affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:39.332 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.329+0000 7f8b6affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:39.332 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.329+0000 7f8b6affd640 10 monclient(hunting): my global_id is 4618 2024-03-29T11:29:39.332 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.329+0000 7f8b6affd640 1 -- 172.21.15.17:0/3242905765 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b580048a0 con 0x7f8b6c0ac5a0 2024-03-29T11:29:39.332 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.329+0000 7f8b6affd640 1 -- 172.21.15.17:0/3242905765 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1947248830 0 0) 0x7f8b580048a0 con 0x7f8b6c0ac5a0 2024-03-29T11:29:39.332 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.329+0000 7f8b6affd640 1 -- 172.21.15.17:0/3242905765 >> v1:172.21.15.17:6790/0 conn(0x7f8b6c14a700 legacy=0x7f8b6c0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:39.332 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.329+0000 7f8b6affd640 1 -- 172.21.15.17:0/3242905765 >> v1:172.21.15.17:6789/0 conn(0x7f8b6c0b0770 legacy=0x7f8b6c149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:39.332 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.329+0000 7f8b6affd640 10 monclient: _finish_hunting 0 2024-03-29T11:29:39.332 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.329+0000 7f8b6affd640 1 monclient: found mon.noname-c 2024-03-29T11:29:39.332 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.329+0000 7f8b6affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:39.333 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.329+0000 7f8b6affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6789/0 2024-03-29T11:29:39.333 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.329+0000 7f8b6affd640 1 -- 172.21.15.17:0/3242905765 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b6c0ac9d0 con 0x7f8b6c0ac5a0 2024-03-29T11:29:39.333 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.329+0000 7f8b6affd640 10 monclient: _finish_auth 0 2024-03-29T11:29:39.333 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.329+0000 7f8b6affd640 10 monclient: _check_auth_tickets 2024-03-29T11:29:39.333 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.329+0000 7f8b6affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:39.333 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.329+0000 7f8b6affd640 1 -- 172.21.15.17:0/3242905765 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f8b54001670 con 0x7f8b6c0ac5a0 2024-03-29T11:29:39.333 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.329+0000 7f8b6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:39.334 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.329+0000 7f8b6affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:39.334 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.329+0000 7f8b6affd640 10 monclient: dump: 2024-03-29T11:29:39.334 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:29:39.334 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:29:39.334 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:39.334 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:39.334 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:39.334 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:39.334 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:29:39.334 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:29:39.334 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:29:39.334 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:39.334 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.329+0000 7f8b7273d640 5 monclient: authenticate success, global_id 4618 2024-03-29T11:29:39.334 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.329+0000 7f8b7273d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:39.334 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.330+0000 7f8b6affd640 1 -- 172.21.15.17:0/3242905765 <== mon.1 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b540017d0 con 0x7f8b6c0ac5a0 2024-03-29T11:29:39.335 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.330+0000 7f8b6affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:39.335 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.330+0000 7f8b6affd640 1 -- 172.21.15.17:0/3242905765 <== mon.1 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f8b54001be0 con 0x7f8b6c0ac5a0 2024-03-29T11:29:39.335 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.330+0000 7f8b6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:39.335 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.330+0000 7f8b6affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:39.335 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.330+0000 7f8b6affd640 10 monclient: dump: 2024-03-29T11:29:39.335 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:29:39.335 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:29:39.335 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:39.335 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:39.335 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:39.335 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:39.335 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:29:39.335 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:29:39.335 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:29:39.335 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:39.336 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.330+0000 7f8b7273d640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:39.336 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.330+0000 7f8b7273d640 4 set_mon_vals no callback set 2024-03-29T11:29:39.336 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.331+0000 7f8b7273d640 10 monclient: shutdown 2024-03-29T11:29:39.336 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.331+0000 7f8b7273d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:39.336 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.331+0000 7f8b7273d640 1 -- 172.21.15.17:0/3242905765 >> v1:172.21.15.94:6789/0 conn(0x7f8b6c0ac5a0 legacy=0x7f8b6c151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:39.336 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.331+0000 7f8b7273d640 1 -- 172.21.15.17:0/3242905765 shutdown_connections 2024-03-29T11:29:39.336 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.331+0000 7f8b7273d640 1 -- 172.21.15.17:0/3242905765 >> 172.21.15.17:0/3242905765 conn(0x7f8b6c06b660 msgr2=0x7f8b6c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:39.336 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.331+0000 7f8b7273d640 1 -- 172.21.15.17:0/3242905765 shutdown_connections 2024-03-29T11:29:39.336 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.331+0000 7f8b7273d640 1 -- 172.21.15.17:0/3242905765 wait complete. 2024-03-29T11:29:39.336 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.332+0000 7f8b7273d640 10 monclient: build_initial_monmap 2024-03-29T11:29:39.336 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.332+0000 7f8b7273d640 10 monclient: monmap: 2024-03-29T11:29:39.336 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:29:39.336 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:39.336 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:29:39.336 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:29:39.337 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:39.337 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:39.337 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:29:39.337 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:29:39.337 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:29:39.337 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:39.337 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.332+0000 7f8b7273d640 1 librados: starting msgr at 2024-03-29T11:29:39.337 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.332+0000 7f8b7273d640 1 librados: starting objecter 2024-03-29T11:29:39.337 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.332+0000 7f8b7273d640 1 Processor -- start 2024-03-29T11:29:39.337 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.333+0000 7f8b7273d640 1 -- start start 2024-03-29T11:29:39.338 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.333+0000 7f8b7273d640 1 librados: setting wanted keys 2024-03-29T11:29:39.338 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.333+0000 7f8b7273d640 1 librados: calling monclient init 2024-03-29T11:29:39.338 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.333+0000 7f8b7273d640 10 monclient: init 2024-03-29T11:29:39.338 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.333+0000 7f8b7273d640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:39.338 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.333+0000 7f8b7273d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:29:39.338 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.333+0000 7f8b7273d640 10 monclient(hunting): picked mon.noname-a con 0x7f8b6c0ac5a0 addr v1:172.21.15.17:6789/0 2024-03-29T11:29:39.338 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.334+0000 7f8b7273d640 10 monclient(hunting): picked mon.noname-b con 0x7f8b6c0b0770 addr v1:172.21.15.94:6789/0 2024-03-29T11:29:39.338 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.334+0000 7f8b6b7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f8b6c0b0770 0x7f8b6c144b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:39570/0 (socket says 172.21.15.17:39570) 2024-03-29T11:29:39.338 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.334+0000 7f8b6b7fe640 1 -- 172.21.15.17:0/213239039 learned_addr learned my addr 172.21.15.17:0/213239039 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:29:39.338 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.334+0000 7f8b7273d640 10 monclient(hunting): picked mon.noname-c con 0x7f8b6c14a700 addr v1:172.21.15.17:6790/0 2024-03-29T11:29:39.338 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.334+0000 7f8b7273d640 1 -- 172.21.15.17:0/213239039 --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b6c1db880 con 0x7f8b6c0ac5a0 2024-03-29T11:29:39.338 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.334+0000 7f8b7273d640 1 -- 172.21.15.17:0/213239039 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b6c145950 con 0x7f8b6c0b0770 2024-03-29T11:29:39.339 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.334+0000 7f8b7273d640 1 -- 172.21.15.17:0/213239039 --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b6c1dc420 con 0x7f8b6c14a700 2024-03-29T11:29:39.339 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.334+0000 7f8b7273d640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:39.339 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.334+0000 7f8b7273d640 10 monclient(hunting): authenticate will time out at 1485.539471s 2024-03-29T11:29:39.339 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.334+0000 7f8b68ff9640 10 client.?.objecter ms_handle_connect 0x7f8b6c0b0770 2024-03-29T11:29:39.339 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.334+0000 7f8b68ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:39.339 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.334+0000 7f8b68ff9640 1 -- 172.21.15.17:0/213239039 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 243621520 0 0) 0x7f8b6c145950 con 0x7f8b6c0b0770 2024-03-29T11:29:39.339 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.334+0000 7f8b68ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:29:39.340 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.334+0000 7f8b68ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:39.340 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.334+0000 7f8b68ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:39.340 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.334+0000 7f8b68ff9640 10 monclient(hunting): my global_id is 4621 2024-03-29T11:29:39.340 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.334+0000 7f8b68ff9640 1 -- 172.21.15.17:0/213239039 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b4c004eb0 con 0x7f8b6c0b0770 2024-03-29T11:29:39.340 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.335+0000 7f8b68ff9640 1 -- 172.21.15.17:0/213239039 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2304096915 0 0) 0x7f8b4c004eb0 con 0x7f8b6c0b0770 2024-03-29T11:29:39.340 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.335+0000 7f8b68ff9640 1 -- 172.21.15.17:0/213239039 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b6c145950 con 0x7f8b6c0b0770 2024-03-29T11:29:39.340 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.335+0000 7f8b68ff9640 10 client.?.objecter ms_handle_connect 0x7f8b6c14a700 2024-03-29T11:29:39.340 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.335+0000 7f8b68ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:39.340 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.335+0000 7f8b68ff9640 10 client.?.objecter ms_handle_connect 0x7f8b6c0ac5a0 2024-03-29T11:29:39.340 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.335+0000 7f8b68ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:39.340 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.335+0000 7f8b68ff9640 1 -- 172.21.15.17:0/213239039 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f8b5c001d10 con 0x7f8b6c0b0770 2024-03-29T11:29:39.340 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.335+0000 7f8b68ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:39.340 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.335+0000 7f8b68ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:39.340 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.335+0000 7f8b68ff9640 10 monclient(hunting): dump: 2024-03-29T11:29:39.341 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:29:39.341 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:29:39.341 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:39.341 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:39.341 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:39.341 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:39.341 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:29:39.341 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:29:39.341 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:29:39.341 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:39.342 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.335+0000 7f8b68ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:39.342 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.336+0000 7f8b68ff9640 1 -- 172.21.15.17:0/213239039 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3350960938 0 0) 0x7f8b6c1dc420 con 0x7f8b6c14a700 2024-03-29T11:29:39.342 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.336+0000 7f8b68ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:29:39.342 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.336+0000 7f8b68ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:39.342 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.336+0000 7f8b68ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:39.342 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.336+0000 7f8b68ff9640 10 monclient(hunting): my global_id is 4676 2024-03-29T11:29:39.342 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.336+0000 7f8b68ff9640 1 -- 172.21.15.17:0/213239039 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b4c004eb0 con 0x7f8b6c14a700 2024-03-29T11:29:39.342 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.336+0000 7f8b68ff9640 1 -- 172.21.15.17:0/213239039 <== mon.1 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1315162078 0 0) 0x7f8b6c145950 con 0x7f8b6c0b0770 2024-03-29T11:29:39.342 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.336+0000 7f8b68ff9640 1 -- 172.21.15.17:0/213239039 >> v1:172.21.15.17:6790/0 conn(0x7f8b6c14a700 legacy=0x7f8b6c145240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:39.342 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.336+0000 7f8b68ff9640 1 -- 172.21.15.17:0/213239039 >> v1:172.21.15.17:6789/0 conn(0x7f8b6c0ac5a0 legacy=0x7f8b6c1dbd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:39.342 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.336+0000 7f8b68ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:29:39.342 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.336+0000 7f8b68ff9640 1 monclient: found mon.b 2024-03-29T11:29:39.342 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.336+0000 7f8b68ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:39.342 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.336+0000 7f8b68ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:29:39.343 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.336+0000 7f8b68ff9640 1 -- 172.21.15.17:0/213239039 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b6c1dc5b0 con 0x7f8b6c0b0770 2024-03-29T11:29:39.343 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.336+0000 7f8b68ff9640 10 monclient: _finish_auth 0 2024-03-29T11:29:39.343 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.336+0000 7f8b68ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:29:39.343 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.336+0000 7f8b68ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:39.343 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.336+0000 7f8b7273d640 5 monclient: authenticate success, global_id 4621 2024-03-29T11:29:39.343 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.336+0000 7f8b7273d640 10 monclient: _renew_subs 2024-03-29T11:29:39.343 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.336+0000 7f8b7273d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:29:39.343 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.336+0000 7f8b7273d640 1 -- 172.21.15.17:0/213239039 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b6c125240 con 0x7f8b6c0b0770 2024-03-29T11:29:39.343 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.336+0000 7f8b7273d640 10 client.4621.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:39.344 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.336+0000 7f8b7273d640 10 monclient: _renew_subs 2024-03-29T11:29:39.344 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.336+0000 7f8b7273d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:29:39.344 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.336+0000 7f8b7273d640 1 -- 172.21.15.17:0/213239039 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b6c123bb0 con 0x7f8b6c0b0770 2024-03-29T11:29:39.344 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.336+0000 7f8b7273d640 1 librados: init done 2024-03-29T11:29:39.344 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.336+0000 7f8b68ff9640 1 -- 172.21.15.17:0/213239039 <== mon.1 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b5c001e70 con 0x7f8b6c0b0770 2024-03-29T11:29:39.344 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.336+0000 7f8b68ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:39.344 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.336+0000 7f8b68ff9640 1 -- 172.21.15.17:0/213239039 <== mon.1 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f8b5c003c40 con 0x7f8b6c0b0770 2024-03-29T11:29:39.344 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.337+0000 7f8b68ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:39.344 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.337+0000 7f8b68ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:39.344 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.337+0000 7f8b68ff9640 10 monclient: dump: 2024-03-29T11:29:39.344 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:29:39.344 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:29:39.344 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:39.344 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:39.345 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:39.345 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:39.345 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:29:39.345 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:29:39.345 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:29:39.345 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:39.345 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.337+0000 7f8b69ffb640 4 set_mon_vals no callback set 2024-03-29T11:29:39.345 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.338+0000 7f8b7273d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:39.345 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.338+0000 7f8b7273d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:39.345 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.338+0000 7f8b7273d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:29:39.345 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.338+0000 7f8b7273d640 1 -- 172.21.15.17:0/213239039 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b50005080 con 0x7f8b6c0b0770 2024-03-29T11:29:39.345 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.339+0000 7f8b68ff9640 1 -- 172.21.15.17:0/213239039 <== mon.1 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f8b5c003da0 con 0x7f8b6c0b0770 2024-03-29T11:29:39.345 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.339+0000 7f8b68ff9640 1 -- 172.21.15.17:0/213239039 <== mon.1 v1:172.21.15.94:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3682049282 0 0) 0x7f8b5c084d90 con 0x7f8b6c0b0770 2024-03-29T11:29:39.345 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.339+0000 7f8b68ff9640 10 client.4621.objecter ms_dispatch 0x7f8b6c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:29:39.345 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.339+0000 7f8b68ff9640 3 client.4621.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:29:39.346 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.339+0000 7f8b68ff9640 3 client.4621.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:29:39.346 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.339+0000 7f8b68ff9640 20 client.4621.objecter dump_active .. 0 homeless 2024-03-29T11:29:39.346 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.340+0000 7f8b6bfff640 10 monclient: get_auth_request con 0x7f8b4c06dd70 auth_method 0 2024-03-29T11:29:39.346 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.340+0000 7f8b68ff9640 10 client.4621.objecter ms_handle_connect 0x7f8b4c06dd70 2024-03-29T11:29:39.346 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.345+0000 7f8b68ff9640 1 -- 172.21.15.17:0/213239039 <== mon.1 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8b5c050670 con 0x7f8b6c0b0770 2024-03-29T11:29:39.347 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.345+0000 7f8b68ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:39.347 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.345+0000 7f8b68ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:39.728 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.736+0000 7f8b7273d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:39.728 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.736+0000 7f8b7273d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:39.729 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.736+0000 7f8b7273d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6789/0 2024-03-29T11:29:39.729 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.736+0000 7f8b7273d640 1 -- 172.21.15.17:0/213239039 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8b50005330 con 0x7f8b6c0b0770 2024-03-29T11:29:39.729 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.737+0000 7f8b68ff9640 1 -- 172.21.15.17:0/213239039 <== mon.1 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 543280943) 0x7f8b5c0525a0 con 0x7f8b6c0b0770 2024-03-29T11:29:39.729 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.737+0000 7f8b68ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:39.729 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.737+0000 7f8b68ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:39.729 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:29:39.730 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":11,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:29:01.208095+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:01.208095+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:39.732 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.741+0000 7f8b7273d640 10 librados: watch_flush enter 2024-03-29T11:29:39.733 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.741+0000 7f8b7273d640 10 librados: watch_flush exit 2024-03-29T11:29:39.733 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.741+0000 7f8b7273d640 20 client.4621.objecter shutdown clearing up homeless session... 2024-03-29T11:29:39.733 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.741+0000 7f8b7273d640 10 client.4621.objecter successfully canceled tick 2024-03-29T11:29:39.733 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.741+0000 7f8b7273d640 1 -- 172.21.15.17:0/213239039 >> v1:172.21.15.17:6816/3236185052 conn(0x7f8b4c06dd70 legacy=0x7f8b4c070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:39.733 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.741+0000 7f8b7273d640 10 monclient: shutdown 2024-03-29T11:29:39.733 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.741+0000 7f8b7273d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:39.733 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.741+0000 7f8b7273d640 1 -- 172.21.15.17:0/213239039 >> v1:172.21.15.94:6789/0 conn(0x7f8b6c0b0770 legacy=0x7f8b6c144b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:39.734 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.742+0000 7f8b7273d640 1 -- 172.21.15.17:0/213239039 shutdown_connections 2024-03-29T11:29:39.734 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.742+0000 7f8b7273d640 1 -- 172.21.15.17:0/213239039 >> 172.21.15.17:0/213239039 conn(0x7f8b6c06b660 msgr2=0x7f8b6c06ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:39.734 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.742+0000 7f8b7273d640 1 -- 172.21.15.17:0/213239039 shutdown_connections 2024-03-29T11:29:39.734 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.742+0000 7f8b7273d640 1 -- 172.21.15.17:0/213239039 wait complete. 2024-03-29T11:29:39.734 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:39.743+0000 7f8b7273d640 1 librados: shutdown 2024-03-29T11:29:39.750 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:46.558 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:46.711 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.718+0000 7f4e63110640 10 monclient: get_monmap_and_config 2024-03-29T11:29:46.711 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.718+0000 7f4e63110640 10 monclient: build_initial_monmap 2024-03-29T11:29:46.711 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.719+0000 7f4e63110640 10 monclient: monmap: 2024-03-29T11:29:46.711 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:29:46.711 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:29:46.711 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:29:46.719988+0000 2024-03-29T11:29:46.711 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:29:46.719988+0000 2024-03-29T11:29:46.711 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:46.711 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:46.711 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:29:46.711 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:29:46.711 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:29:46.712 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:46.714 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.721+0000 7f4e63110640 1 Processor -- start 2024-03-29T11:29:46.714 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.721+0000 7f4e63110640 1 -- start start 2024-03-29T11:29:46.714 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.721+0000 7f4e63110640 10 monclient: init 2024-03-29T11:29:46.714 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.722+0000 7f4e63110640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:46.714 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.722+0000 7f4e63110640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:29:46.714 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.722+0000 7f4e63110640 10 monclient(hunting): picked mon.noname-b con 0x7f4e5c1422b0 addr v1:172.21.15.17:6790/0 2024-03-29T11:29:46.714 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.722+0000 7f4e63110640 10 monclient(hunting): picked mon.noname-a con 0x7f4e5c142690 addr v1:172.21.15.17:6789/0 2024-03-29T11:29:46.714 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.722+0000 7f4e63110640 10 monclient(hunting): picked mon.noname-c con 0x7f4e5c0b17d0 addr v1:172.21.15.94:6789/0 2024-03-29T11:29:46.714 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.722+0000 7f4e63110640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e5c142c80 con 0x7f4e5c142690 2024-03-29T11:29:46.714 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.722+0000 7f4e63110640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e5c0aae40 con 0x7f4e5c0b17d0 2024-03-29T11:29:46.714 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.722+0000 7f4e63110640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e5c13fbd0 con 0x7f4e5c1422b0 2024-03-29T11:29:46.714 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.722+0000 7f4e63110640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:46.715 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.722+0000 7f4e63110640 10 monclient(hunting): authenticate will time out at 1492.927055s 2024-03-29T11:29:46.718 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.726+0000 7f4e616ad640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f4e5c0b17d0 0x7f4e5c0b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:55573/0 (socket says 172.21.15.17:55573) 2024-03-29T11:29:46.718 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.726+0000 7f4e616ad640 1 -- 172.21.15.17:0/3008006052 learned_addr learned my addr 172.21.15.17:0/3008006052 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:29:46.719 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.727+0000 7f4e5b7fe640 1 -- 172.21.15.17:0/3008006052 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3338850299 0 0) 0x7f4e5c0aae40 con 0x7f4e5c0b17d0 2024-03-29T11:29:46.719 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.727+0000 7f4e5b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:46.719 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.727+0000 7f4e5b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:46.719 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.727+0000 7f4e5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:46.719 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.727+0000 7f4e5b7fe640 10 monclient(hunting): my global_id is 4624 2024-03-29T11:29:46.720 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.727+0000 7f4e5b7fe640 1 -- 172.21.15.17:0/3008006052 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e44003520 con 0x7f4e5c0b17d0 2024-03-29T11:29:46.720 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.727+0000 7f4e5b7fe640 1 -- 172.21.15.17:0/3008006052 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2259164532 0 0) 0x7f4e5c142c80 con 0x7f4e5c142690 2024-03-29T11:29:46.720 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.727+0000 7f4e5b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:46.720 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.727+0000 7f4e5b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:46.720 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.727+0000 7f4e5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:46.720 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.727+0000 7f4e5b7fe640 10 monclient(hunting): my global_id is 4569 2024-03-29T11:29:46.720 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.727+0000 7f4e5b7fe640 1 -- 172.21.15.17:0/3008006052 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e5c0aae40 con 0x7f4e5c142690 2024-03-29T11:29:46.720 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.727+0000 7f4e5b7fe640 1 -- 172.21.15.17:0/3008006052 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1871332893 0 0) 0x7f4e44003520 con 0x7f4e5c0b17d0 2024-03-29T11:29:46.720 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.727+0000 7f4e5b7fe640 1 -- 172.21.15.17:0/3008006052 >> v1:172.21.15.17:6790/0 conn(0x7f4e5c1422b0 legacy=0x7f4e5c13f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:46.725 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.729+0000 7f4e5b7fe640 1 -- 172.21.15.17:0/3008006052 >> v1:172.21.15.17:6789/0 conn(0x7f4e5c142690 legacy=0x7f4e5c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:46.725 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.729+0000 7f4e5b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:29:46.726 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.729+0000 7f4e5b7fe640 1 monclient: found mon.noname-c 2024-03-29T11:29:46.726 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.729+0000 7f4e5b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:46.726 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.729+0000 7f4e5b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6789/0 2024-03-29T11:29:46.726 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.729+0000 7f4e5b7fe640 1 -- 172.21.15.17:0/3008006052 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e5c151940 con 0x7f4e5c0b17d0 2024-03-29T11:29:46.726 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.729+0000 7f4e5b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:29:46.726 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.729+0000 7f4e5b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:29:46.727 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.729+0000 7f4e5b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:46.727 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.730+0000 7f4e63110640 5 monclient: authenticate success, global_id 4624 2024-03-29T11:29:46.727 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.730+0000 7f4e63110640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:46.727 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.730+0000 7f4e5b7fe640 1 -- 172.21.15.17:0/3008006052 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f4e50001690 con 0x7f4e5c0b17d0 2024-03-29T11:29:46.727 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.730+0000 7f4e5b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:46.727 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.730+0000 7f4e5b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:46.728 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.730+0000 7f4e5b7fe640 10 monclient: dump: 2024-03-29T11:29:46.728 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:29:46.728 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:29:46.728 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:46.728 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:46.728 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:46.728 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:46.728 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:29:46.729 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:29:46.729 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:29:46.729 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:46.729 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.730+0000 7f4e63110640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:46.729 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.730+0000 7f4e5b7fe640 1 -- 172.21.15.17:0/3008006052 <== mon.1 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e500018b0 con 0x7f4e5c0b17d0 2024-03-29T11:29:46.729 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.730+0000 7f4e5b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:46.729 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.730+0000 7f4e63110640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:46.730 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.730+0000 7f4e63110640 4 set_mon_vals no callback set 2024-03-29T11:29:46.730 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.731+0000 7f4e63110640 10 monclient: shutdown 2024-03-29T11:29:46.730 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.731+0000 7f4e63110640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:46.730 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.731+0000 7f4e63110640 1 -- 172.21.15.17:0/3008006052 >> v1:172.21.15.94:6789/0 conn(0x7f4e5c0b17d0 legacy=0x7f4e5c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:46.730 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.731+0000 7f4e63110640 1 -- 172.21.15.17:0/3008006052 shutdown_connections 2024-03-29T11:29:46.730 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.731+0000 7f4e63110640 1 -- 172.21.15.17:0/3008006052 >> 172.21.15.17:0/3008006052 conn(0x7f4e5c066a00 msgr2=0x7f4e5c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:46.732 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.734+0000 7f4e63110640 1 -- 172.21.15.17:0/3008006052 shutdown_connections 2024-03-29T11:29:46.732 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.734+0000 7f4e63110640 1 -- 172.21.15.17:0/3008006052 wait complete. 2024-03-29T11:29:46.732 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.735+0000 7f4e63110640 10 monclient: build_initial_monmap 2024-03-29T11:29:46.732 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.735+0000 7f4e63110640 10 monclient: monmap: 2024-03-29T11:29:46.732 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:29:46.733 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:46.733 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:29:46.733 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:29:46.733 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:46.733 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:46.733 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:29:46.733 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:29:46.733 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:29:46.733 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:46.733 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.736+0000 7f4e63110640 1 librados: starting msgr at 2024-03-29T11:29:46.733 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.736+0000 7f4e63110640 1 librados: starting objecter 2024-03-29T11:29:46.733 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.736+0000 7f4e63110640 1 Processor -- start 2024-03-29T11:29:46.733 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.736+0000 7f4e63110640 1 -- start start 2024-03-29T11:29:46.734 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.736+0000 7f4e63110640 1 librados: setting wanted keys 2024-03-29T11:29:46.734 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.736+0000 7f4e63110640 1 librados: calling monclient init 2024-03-29T11:29:46.734 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.737+0000 7f4e63110640 10 monclient: init 2024-03-29T11:29:46.734 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.738+0000 7f4e63110640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:46.734 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.738+0000 7f4e63110640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:29:46.735 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.741+0000 7f4e63110640 10 monclient(hunting): picked mon.noname-a con 0x7f4e5c1422b0 addr v1:172.21.15.17:6789/0 2024-03-29T11:29:46.735 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.741+0000 7f4e60eac640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f4e5c1422b0 0x7f4e5c141850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:40945/0 (socket says 172.21.15.17:40945) 2024-03-29T11:29:46.735 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.741+0000 7f4e60eac640 1 -- 172.21.15.17:0/1186351452 learned_addr learned my addr 172.21.15.17:0/1186351452 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:29:46.735 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.742+0000 7f4e63110640 10 monclient(hunting): picked mon.noname-b con 0x7f4e5c142690 addr v1:172.21.15.94:6789/0 2024-03-29T11:29:46.735 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.742+0000 7f4e597fa640 10 client.?.objecter ms_handle_connect 0x7f4e5c1422b0 2024-03-29T11:29:46.735 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.742+0000 7f4e597fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:46.735 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.742+0000 7f4e63110640 10 monclient(hunting): picked mon.noname-c con 0x7f4e5c1295b0 addr v1:172.21.15.17:6790/0 2024-03-29T11:29:46.735 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.742+0000 7f4e63110640 1 -- 172.21.15.17:0/1186351452 --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e5c141140 con 0x7f4e5c1422b0 2024-03-29T11:29:46.735 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.742+0000 7f4e63110640 1 -- 172.21.15.17:0/1186351452 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e5c0b5a10 con 0x7f4e5c142690 2024-03-29T11:29:46.735 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.742+0000 7f4e63110640 1 -- 172.21.15.17:0/1186351452 --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e5c0b6850 con 0x7f4e5c1295b0 2024-03-29T11:29:46.735 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.742+0000 7f4e63110640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:46.735 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.742+0000 7f4e63110640 10 monclient(hunting): authenticate will time out at 1492.947736s 2024-03-29T11:29:46.737 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.745+0000 7f4e597fa640 1 -- 172.21.15.17:0/1186351452 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2079839299 0 0) 0x7f4e5c141140 con 0x7f4e5c1422b0 2024-03-29T11:29:46.737 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.745+0000 7f4e597fa640 10 monclient(hunting): _negotiate 2024-03-29T11:29:46.737 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.745+0000 7f4e597fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:46.737 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.745+0000 7f4e597fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:46.737 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.745+0000 7f4e597fa640 10 monclient(hunting): my global_id is 4572 2024-03-29T11:29:46.737 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.745+0000 7f4e597fa640 1 -- 172.21.15.17:0/1186351452 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e48004eb0 con 0x7f4e5c1422b0 2024-03-29T11:29:46.738 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.746+0000 7f4e597fa640 1 -- 172.21.15.17:0/1186351452 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1871728033 0 0) 0x7f4e48004eb0 con 0x7f4e5c1422b0 2024-03-29T11:29:46.738 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.746+0000 7f4e597fa640 1 -- 172.21.15.17:0/1186351452 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e5c141140 con 0x7f4e5c1422b0 2024-03-29T11:29:46.738 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.746+0000 7f4e597fa640 1 -- 172.21.15.17:0/1186351452 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f4e4c001960 con 0x7f4e5c1422b0 2024-03-29T11:29:46.738 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.746+0000 7f4e597fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:46.738 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.746+0000 7f4e597fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:46.738 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.746+0000 7f4e597fa640 10 monclient(hunting): dump: 2024-03-29T11:29:46.738 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:29:46.738 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:29:46.738 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:46.738 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:46.738 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:46.738 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:46.738 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:29:46.739 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:29:46.739 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:29:46.739 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:46.739 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.746+0000 7f4e597fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:46.739 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.746+0000 7f4e597fa640 1 -- 172.21.15.17:0/1186351452 <== mon.0 v1:172.21.15.17:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 844625853 0 0) 0x7f4e5c141140 con 0x7f4e5c1422b0 2024-03-29T11:29:46.739 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.746+0000 7f4e597fa640 1 -- 172.21.15.17:0/1186351452 >> v1:172.21.15.17:6790/0 conn(0x7f4e5c1295b0 legacy=0x7f4e5c1e1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:46.739 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.746+0000 7f4e597fa640 1 -- 172.21.15.17:0/1186351452 >> v1:172.21.15.94:6789/0 conn(0x7f4e5c142690 legacy=0x7f4e5c0b72c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:29:46.739 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.746+0000 7f4e597fa640 10 monclient: _finish_hunting 0 2024-03-29T11:29:46.739 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.746+0000 7f4e597fa640 1 monclient: found mon.a 2024-03-29T11:29:46.739 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.746+0000 7f4e597fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:46.740 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.746+0000 7f4e597fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:29:46.740 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.746+0000 7f4e597fa640 1 -- 172.21.15.17:0/1186351452 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e5c0b6b30 con 0x7f4e5c1422b0 2024-03-29T11:29:46.740 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.746+0000 7f4e597fa640 10 monclient: _finish_auth 0 2024-03-29T11:29:46.740 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.746+0000 7f4e597fa640 10 monclient: _check_auth_tickets 2024-03-29T11:29:46.740 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.746+0000 7f4e597fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:46.740 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.746+0000 7f4e63110640 5 monclient: authenticate success, global_id 4572 2024-03-29T11:29:46.740 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.746+0000 7f4e63110640 10 monclient: _renew_subs 2024-03-29T11:29:46.740 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.746+0000 7f4e63110640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:29:46.740 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.746+0000 7f4e63110640 1 -- 172.21.15.17:0/1186351452 --> v1:172.21.15.17:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e5c140dd0 con 0x7f4e5c1422b0 2024-03-29T11:29:46.740 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.746+0000 7f4e63110640 10 client.4572.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:46.740 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.747+0000 7f4e63110640 10 monclient: _renew_subs 2024-03-29T11:29:46.740 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.747+0000 7f4e63110640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:29:46.740 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.747+0000 7f4e63110640 1 -- 172.21.15.17:0/1186351452 --> v1:172.21.15.17:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e5c1e3580 con 0x7f4e5c1422b0 2024-03-29T11:29:46.740 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.747+0000 7f4e63110640 1 librados: init done 2024-03-29T11:29:46.740 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.747+0000 7f4e597fa640 1 -- 172.21.15.17:0/1186351452 <== mon.0 v1:172.21.15.17:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e4c001ac0 con 0x7f4e5c1422b0 2024-03-29T11:29:46.741 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.747+0000 7f4e597fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:46.741 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.747+0000 7f4e597fa640 1 -- 172.21.15.17:0/1186351452 <== mon.0 v1:172.21.15.17:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f4e4c003a40 con 0x7f4e5c1422b0 2024-03-29T11:29:46.741 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.747+0000 7f4e597fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:46.741 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.747+0000 7f4e597fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:46.741 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.747+0000 7f4e597fa640 10 monclient: dump: 2024-03-29T11:29:46.741 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:29:46.741 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:29:46.741 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:46.741 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:46.741 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:46.741 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:46.741 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:29:46.741 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:29:46.741 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:29:46.741 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:46.743 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.750+0000 7f4e5affd640 4 set_mon_vals no callback set 2024-03-29T11:29:46.743 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.750+0000 7f4e597fa640 1 -- 172.21.15.17:0/1186351452 <== mon.0 v1:172.21.15.17:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f4e4c003ba0 con 0x7f4e5c1422b0 2024-03-29T11:29:46.743 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.751+0000 7f4e63110640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:46.743 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.751+0000 7f4e63110640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:46.743 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.751+0000 7f4e63110640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:29:46.743 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.751+0000 7f4e63110640 1 -- 172.21.15.17:0/1186351452 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e3c005080 con 0x7f4e5c1422b0 2024-03-29T11:29:46.747 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.751+0000 7f4e597fa640 1 -- 172.21.15.17:0/1186351452 <== mon.0 v1:172.21.15.17:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3682049282 0 0) 0x7f4e4c084b40 con 0x7f4e5c1422b0 2024-03-29T11:29:46.747 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.754+0000 7f4e597fa640 10 client.4572.objecter ms_dispatch 0x7f4e5c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:29:46.747 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.754+0000 7f4e597fa640 3 client.4572.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:29:46.747 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.754+0000 7f4e597fa640 3 client.4572.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:29:46.747 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.754+0000 7f4e597fa640 20 client.4572.objecter dump_active .. 0 homeless 2024-03-29T11:29:46.747 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.755+0000 7f4e5bfff640 10 monclient: get_auth_request con 0x7f4e4806dcc0 auth_method 0 2024-03-29T11:29:46.747 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.755+0000 7f4e597fa640 10 client.4572.objecter ms_handle_connect 0x7f4e4806dcc0 2024-03-29T11:29:46.754 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.763+0000 7f4e597fa640 1 -- 172.21.15.17:0/1186351452 <== mon.0 v1:172.21.15.17:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4e4c050420 con 0x7f4e5c1422b0 2024-03-29T11:29:46.754 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.763+0000 7f4e597fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:46.754 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:46.763+0000 7f4e597fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:47.122 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:47.130+0000 7f4e63110640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:47.122 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:47.130+0000 7f4e63110640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:47.122 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:47.130+0000 7f4e63110640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:29:47.122 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:47.130+0000 7f4e63110640 1 -- 172.21.15.17:0/1186351452 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e3c005330 con 0x7f4e5c1422b0 2024-03-29T11:29:47.123 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:47.131+0000 7f4e597fa640 1 -- 172.21.15.17:0/1186351452 <== mon.0 v1:172.21.15.17:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 543280943) 0x7f4e4c052350 con 0x7f4e5c1422b0 2024-03-29T11:29:47.123 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:47.131+0000 7f4e597fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:47.123 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:47.131+0000 7f4e597fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:47.123 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:29:47.123 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":11,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:29:01.208095+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:01.208095+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:47.126 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:47.134+0000 7f4e63110640 10 librados: watch_flush enter 2024-03-29T11:29:47.126 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:47.134+0000 7f4e63110640 10 librados: watch_flush exit 2024-03-29T11:29:47.126 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:47.134+0000 7f4e63110640 20 client.4572.objecter shutdown clearing up homeless session... 2024-03-29T11:29:47.126 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:47.134+0000 7f4e63110640 10 client.4572.objecter successfully canceled tick 2024-03-29T11:29:47.127 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:47.135+0000 7f4e63110640 1 -- 172.21.15.17:0/1186351452 >> v1:172.21.15.17:6816/3236185052 conn(0x7f4e4806dcc0 legacy=0x7f4e48070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:47.127 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:47.135+0000 7f4e63110640 10 monclient: shutdown 2024-03-29T11:29:47.127 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:47.135+0000 7f4e63110640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:47.127 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:47.135+0000 7f4e63110640 1 -- 172.21.15.17:0/1186351452 >> v1:172.21.15.17:6789/0 conn(0x7f4e5c1422b0 legacy=0x7f4e5c141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:47.127 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:47.135+0000 7f4e63110640 1 -- 172.21.15.17:0/1186351452 shutdown_connections 2024-03-29T11:29:47.127 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:47.135+0000 7f4e63110640 1 -- 172.21.15.17:0/1186351452 >> 172.21.15.17:0/1186351452 conn(0x7f4e5c066a00 msgr2=0x7f4e5c129ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:47.128 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:47.136+0000 7f4e63110640 1 -- 172.21.15.17:0/1186351452 shutdown_connections 2024-03-29T11:29:47.128 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:47.136+0000 7f4e63110640 1 -- 172.21.15.17:0/1186351452 wait complete. 2024-03-29T11:29:47.128 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:47.136+0000 7f4e63110640 1 librados: shutdown 2024-03-29T11:29:47.142 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:53.950 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:54.102 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.108+0000 7f880266a640 10 monclient: get_monmap_and_config 2024-03-29T11:29:54.102 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.108+0000 7f880266a640 10 monclient: build_initial_monmap 2024-03-29T11:29:54.102 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.108+0000 7f880266a640 10 monclient: monmap: 2024-03-29T11:29:54.102 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:29:54.102 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:29:54.102 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:29:54.109712+0000 2024-03-29T11:29:54.102 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:29:54.109712+0000 2024-03-29T11:29:54.102 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:54.102 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:54.102 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:29:54.103 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-c 2024-03-29T11:29:54.103 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-b 2024-03-29T11:29:54.103 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:54.103 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.109+0000 7f880266a640 1 Processor -- start 2024-03-29T11:29:54.103 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.109+0000 7f880266a640 1 -- start start 2024-03-29T11:29:54.103 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.109+0000 7f880266a640 10 monclient: init 2024-03-29T11:29:54.110 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.110+0000 7f880266a640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:54.111 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.110+0000 7f880266a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:29:54.111 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.110+0000 7f880266a640 10 monclient(hunting): picked mon.noname-c con 0x7f87fc0ae250 addr v1:172.21.15.94:6789/0 2024-03-29T11:29:54.111 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.110+0000 7f880266a640 10 monclient(hunting): picked mon.noname-a con 0x7f87fc0a9720 addr v1:172.21.15.17:6789/0 2024-03-29T11:29:54.111 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.110+0000 7f880266a640 10 monclient(hunting): picked mon.noname-b con 0x7f87fc14bba0 addr v1:172.21.15.17:6790/0 2024-03-29T11:29:54.111 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.110+0000 7f880266a640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87fc0a9b80 con 0x7f87fc0a9720 2024-03-29T11:29:54.111 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.110+0000 7f880266a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87fc0aa7b0 con 0x7f87fc0ae250 2024-03-29T11:29:54.111 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.110+0000 7f880266a640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87fc0afbf0 con 0x7f87fc14bba0 2024-03-29T11:29:54.111 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.110+0000 7f880266a640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:54.111 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.110+0000 7f880266a640 10 monclient(hunting): authenticate will time out at 1500.315809s 2024-03-29T11:29:54.111 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.111+0000 7f87fb7fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f87fc0a9720 0x7f87fc13f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:54938/0 (socket says 172.21.15.17:54938) 2024-03-29T11:29:54.112 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.111+0000 7f87fb7fe640 1 -- 172.21.15.17:0/4112525229 learned_addr learned my addr 172.21.15.17:0/4112525229 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:29:54.112 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.114+0000 7f87faffd640 1 -- 172.21.15.17:0/4112525229 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3010581469 0 0) 0x7f87fc0a9b80 con 0x7f87fc0a9720 2024-03-29T11:29:54.112 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.114+0000 7f87faffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:54.112 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.114+0000 7f87faffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:54.112 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.114+0000 7f87faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:54.112 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.114+0000 7f87faffd640 10 monclient(hunting): my global_id is 4575 2024-03-29T11:29:54.112 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.114+0000 7f87faffd640 1 -- 172.21.15.17:0/4112525229 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87e8003520 con 0x7f87fc0a9720 2024-03-29T11:29:54.112 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.114+0000 7f87faffd640 1 -- 172.21.15.17:0/4112525229 <== mon.2 v1:172.21.15.17:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 947643933 0 0) 0x7f87fc0afbf0 con 0x7f87fc14bba0 2024-03-29T11:29:54.112 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.114+0000 7f87faffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:54.112 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.114+0000 7f87faffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:54.112 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.114+0000 7f87faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:54.113 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.114+0000 7f87faffd640 10 monclient(hunting): my global_id is 4679 2024-03-29T11:29:54.113 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.114+0000 7f87faffd640 1 -- 172.21.15.17:0/4112525229 --> v1:172.21.15.17:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87fc0a9b80 con 0x7f87fc14bba0 2024-03-29T11:29:54.113 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.114+0000 7f87faffd640 1 -- 172.21.15.17:0/4112525229 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 424188246 0 0) 0x7f87fc0aa7b0 con 0x7f87fc0ae250 2024-03-29T11:29:54.113 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.114+0000 7f87faffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:54.113 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.114+0000 7f87faffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:54.113 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.114+0000 7f87faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:54.113 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.114+0000 7f87faffd640 10 monclient(hunting): my global_id is 4627 2024-03-29T11:29:54.113 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.114+0000 7f87faffd640 1 -- 172.21.15.17:0/4112525229 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87fc0afbf0 con 0x7f87fc0ae250 2024-03-29T11:29:54.113 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.114+0000 7f87faffd640 1 -- 172.21.15.17:0/4112525229 <== mon.2 v1:172.21.15.17:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1501701039 0 0) 0x7f87fc0a9b80 con 0x7f87fc14bba0 2024-03-29T11:29:54.113 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.115+0000 7f87faffd640 1 -- 172.21.15.17:0/4112525229 >> v1:172.21.15.94:6789/0 conn(0x7f87fc0ae250 legacy=0x7f87fc142b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:54.113 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.115+0000 7f87faffd640 1 -- 172.21.15.17:0/4112525229 >> v1:172.21.15.17:6789/0 conn(0x7f87fc0a9720 legacy=0x7f87fc13f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:54.114 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.115+0000 7f87faffd640 10 monclient: _finish_hunting 0 2024-03-29T11:29:54.114 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.115+0000 7f87faffd640 1 monclient: found mon.noname-b 2024-03-29T11:29:54.114 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.115+0000 7f87faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:54.114 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.115+0000 7f87faffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.17:6790/0 2024-03-29T11:29:54.114 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.115+0000 7f87faffd640 1 -- 172.21.15.17:0/4112525229 --> v1:172.21.15.17:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87fc143280 con 0x7f87fc14bba0 2024-03-29T11:29:54.114 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.115+0000 7f87faffd640 10 monclient: _finish_auth 0 2024-03-29T11:29:54.114 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.115+0000 7f87faffd640 10 monclient: _check_auth_tickets 2024-03-29T11:29:54.114 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.115+0000 7f87faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:54.114 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.115+0000 7f87faffd640 1 -- 172.21.15.17:0/4112525229 <== mon.2 v1:172.21.15.17:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f87ec0019f0 con 0x7f87fc14bba0 2024-03-29T11:29:54.114 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.115+0000 7f87faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:54.114 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.115+0000 7f87faffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:54.114 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.115+0000 7f87faffd640 10 monclient: dump: 2024-03-29T11:29:54.114 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:29:54.115 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:29:54.115 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:54.115 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:54.115 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:54.115 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:54.115 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:29:54.115 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:29:54.115 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:29:54.115 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:54.115 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.115+0000 7f880266a640 5 monclient: authenticate success, global_id 4679 2024-03-29T11:29:54.115 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.115+0000 7f880266a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:54.115 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.115+0000 7f87faffd640 1 -- 172.21.15.17:0/4112525229 <== mon.2 v1:172.21.15.17:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f87ec001b50 con 0x7f87fc14bba0 2024-03-29T11:29:54.115 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.115+0000 7f87faffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:54.116 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.115+0000 7f87faffd640 1 -- 172.21.15.17:0/4112525229 <== mon.2 v1:172.21.15.17:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f87ec001ee0 con 0x7f87fc14bba0 2024-03-29T11:29:54.116 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.115+0000 7f87faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:54.116 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.115+0000 7f87faffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:54.116 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.115+0000 7f87faffd640 10 monclient: dump: 2024-03-29T11:29:54.116 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:29:54.117 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:29:54.117 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:54.117 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:54.117 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:54.117 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:54.117 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:29:54.117 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:29:54.117 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:29:54.117 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:54.117 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.115+0000 7f880266a640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:54.117 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.115+0000 7f880266a640 4 set_mon_vals no callback set 2024-03-29T11:29:54.117 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.116+0000 7f880266a640 10 monclient: shutdown 2024-03-29T11:29:54.118 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.116+0000 7f880266a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:54.118 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.116+0000 7f880266a640 1 -- 172.21.15.17:0/4112525229 >> v1:172.21.15.17:6790/0 conn(0x7f87fc14bba0 legacy=0x7f87fc14df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:54.118 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.116+0000 7f880266a640 1 -- 172.21.15.17:0/4112525229 shutdown_connections 2024-03-29T11:29:54.118 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.116+0000 7f880266a640 1 -- 172.21.15.17:0/4112525229 >> 172.21.15.17:0/4112525229 conn(0x7f87fc0b35e0 msgr2=0x7f87fc0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:54.118 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.116+0000 7f880266a640 1 -- 172.21.15.17:0/4112525229 shutdown_connections 2024-03-29T11:29:54.118 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.116+0000 7f880266a640 1 -- 172.21.15.17:0/4112525229 wait complete. 2024-03-29T11:29:54.121 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.117+0000 7f880266a640 10 monclient: build_initial_monmap 2024-03-29T11:29:54.121 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.117+0000 7f880266a640 10 monclient: monmap: 2024-03-29T11:29:54.121 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-03-29T11:29:54.121 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:54.121 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-03-29T11:29:54.121 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-03-29T11:29:54.122 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:54.122 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:54.122 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a 2024-03-29T11:29:54.122 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.noname-b 2024-03-29T11:29:54.122 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.noname-c 2024-03-29T11:29:54.122 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:54.122 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.117+0000 7f880266a640 1 librados: starting msgr at 2024-03-29T11:29:54.122 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.117+0000 7f880266a640 1 librados: starting objecter 2024-03-29T11:29:54.122 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.117+0000 7f880266a640 1 Processor -- start 2024-03-29T11:29:54.122 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.120+0000 7f880266a640 1 -- start start 2024-03-29T11:29:54.122 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.120+0000 7f880266a640 1 librados: setting wanted keys 2024-03-29T11:29:54.122 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.120+0000 7f880266a640 1 librados: calling monclient init 2024-03-29T11:29:54.123 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.120+0000 7f880266a640 10 monclient: init 2024-03-29T11:29:54.123 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.121+0000 7f880266a640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:54.123 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.121+0000 7f880266a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:29:54.123 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.121+0000 7f880266a640 10 monclient(hunting): picked mon.noname-b con 0x7f87fc0a9720 addr v1:172.21.15.94:6789/0 2024-03-29T11:29:54.123 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.121+0000 7f880266a640 10 monclient(hunting): picked mon.noname-c con 0x7f87fc0ae250 addr v1:172.21.15.17:6790/0 2024-03-29T11:29:54.123 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.121+0000 7f880266a640 10 monclient(hunting): picked mon.noname-a con 0x7f87fc14bba0 addr v1:172.21.15.17:6789/0 2024-03-29T11:29:54.123 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.121+0000 7f880266a640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87fc06fd80 con 0x7f87fc14bba0 2024-03-29T11:29:54.123 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.121+0000 7f880266a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87fc13c110 con 0x7f87fc0a9720 2024-03-29T11:29:54.123 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.121+0000 7f880266a640 1 -- --> v1:172.21.15.17:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87fc13bd10 con 0x7f87fc0ae250 2024-03-29T11:29:54.123 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.121+0000 7f880266a640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:54.123 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.121+0000 7f880266a640 10 monclient(hunting): authenticate will time out at 1500.326678s 2024-03-29T11:29:54.123 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.126+0000 7f87fbfff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f87fc0a9720 0x7f87fc1e4280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.17:60777/0 (socket says 172.21.15.17:60777) 2024-03-29T11:29:54.124 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.126+0000 7f87fbfff640 1 -- 172.21.15.17:0/783890441 learned_addr learned my addr 172.21.15.17:0/783890441 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-03-29T11:29:54.124 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.126+0000 7f87f8ff9640 10 client.?.objecter ms_handle_connect 0x7f87fc0a9720 2024-03-29T11:29:54.124 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.126+0000 7f87f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:54.124 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.126+0000 7f87f8ff9640 10 client.?.objecter ms_handle_connect 0x7f87fc14bba0 2024-03-29T11:29:54.124 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.126+0000 7f87f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:54.125 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.128+0000 7f87f8ff9640 10 client.?.objecter ms_handle_connect 0x7f87fc0ae250 2024-03-29T11:29:54.125 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.128+0000 7f87f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:54.125 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.128+0000 7f87f8ff9640 1 -- 172.21.15.17:0/783890441 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1763158285 0 0) 0x7f87fc143280 con 0x7f87fc14bba0 2024-03-29T11:29:54.125 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.128+0000 7f87f8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:29:54.126 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.128+0000 7f87f8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:54.126 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.128+0000 7f87f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:54.126 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.128+0000 7f87f8ff9640 10 monclient(hunting): my global_id is 4578 2024-03-29T11:29:54.126 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.128+0000 7f87f8ff9640 1 -- 172.21.15.17:0/783890441 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87dc004eb0 con 0x7f87fc14bba0 2024-03-29T11:29:54.126 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.128+0000 7f87f8ff9640 1 -- 172.21.15.17:0/783890441 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2790339431 0 0) 0x7f87fc13c110 con 0x7f87fc0a9720 2024-03-29T11:29:54.126 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.128+0000 7f87f8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:29:54.126 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.128+0000 7f87f8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:54.126 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.128+0000 7f87f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:54.126 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.128+0000 7f87f8ff9640 10 monclient(hunting): my global_id is 4630 2024-03-29T11:29:54.126 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.128+0000 7f87f8ff9640 1 -- 172.21.15.17:0/783890441 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87fc143280 con 0x7f87fc0a9720 2024-03-29T11:29:54.126 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.128+0000 7f87f8ff9640 1 -- 172.21.15.17:0/783890441 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1888036182 0 0) 0x7f87dc004eb0 con 0x7f87fc14bba0 2024-03-29T11:29:54.126 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.128+0000 7f87f8ff9640 1 -- 172.21.15.17:0/783890441 --> v1:172.21.15.17:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87fc13c110 con 0x7f87fc14bba0 2024-03-29T11:29:54.127 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.128+0000 7f87f8ff9640 1 -- 172.21.15.17:0/783890441 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f87ec001cc0 con 0x7f87fc14bba0 2024-03-29T11:29:54.127 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.128+0000 7f87f8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:54.127 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.128+0000 7f87f8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:54.127 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.128+0000 7f87f8ff9640 10 monclient(hunting): dump: 2024-03-29T11:29:54.127 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:29:54.127 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:29:54.127 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:54.127 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:54.127 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:54.127 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:54.127 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:29:54.127 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:29:54.127 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:29:54.128 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:54.128 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.128+0000 7f87f8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:54.128 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.128+0000 7f87f8ff9640 1 -- 172.21.15.17:0/783890441 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 336715380 0 0) 0x7f87fc143280 con 0x7f87fc0a9720 2024-03-29T11:29:54.128 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.128+0000 7f87f8ff9640 1 -- 172.21.15.17:0/783890441 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87dc004eb0 con 0x7f87fc0a9720 2024-03-29T11:29:54.128 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.128+0000 7f87f8ff9640 1 -- 172.21.15.17:0/783890441 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f87f40018f0 con 0x7f87fc0a9720 2024-03-29T11:29:54.128 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.128+0000 7f87f8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:54.128 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.128+0000 7f87f8ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:54.128 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.128+0000 7f87f8ff9640 10 monclient(hunting): dump: 2024-03-29T11:29:54.128 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:29:54.128 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:29:54.128 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:54.128 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:54.128 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:54.129 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:54.129 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:29:54.129 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:29:54.129 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:29:54.129 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:54.129 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.128+0000 7f87f8ff9640 1 -- 172.21.15.17:0/783890441 <== mon.0 v1:172.21.15.17:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1736430462 0 0) 0x7f87fc13c110 con 0x7f87fc14bba0 2024-03-29T11:29:54.129 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.128+0000 7f87f8ff9640 1 -- 172.21.15.17:0/783890441 >> v1:172.21.15.17:6790/0 conn(0x7f87fc0ae250 legacy=0x7f87fc071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:54.129 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.128+0000 7f87f8ff9640 1 -- 172.21.15.17:0/783890441 >> v1:172.21.15.94:6789/0 conn(0x7f87fc0a9720 legacy=0x7f87fc1e4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:54.129 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.128+0000 7f87f8ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:29:54.129 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.128+0000 7f87f8ff9640 1 monclient: found mon.a 2024-03-29T11:29:54.129 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.128+0000 7f87f8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:54.129 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.128+0000 7f87f8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:29:54.129 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.128+0000 7f87f8ff9640 1 -- 172.21.15.17:0/783890441 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87fc13c7b0 con 0x7f87fc14bba0 2024-03-29T11:29:54.130 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.128+0000 7f87f8ff9640 10 monclient: _finish_auth 0 2024-03-29T11:29:54.130 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.128+0000 7f87f8ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:29:54.130 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.128+0000 7f87f8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:54.130 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.128+0000 7f880266a640 5 monclient: authenticate success, global_id 4578 2024-03-29T11:29:54.130 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.128+0000 7f880266a640 10 monclient: _renew_subs 2024-03-29T11:29:54.130 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.128+0000 7f880266a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:29:54.130 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.128+0000 7f880266a640 1 -- 172.21.15.17:0/783890441 --> v1:172.21.15.17:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87fc1e4990 con 0x7f87fc14bba0 2024-03-29T11:29:54.130 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.129+0000 7f880266a640 10 client.4578.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:54.130 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.129+0000 7f880266a640 10 monclient: _renew_subs 2024-03-29T11:29:54.130 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.129+0000 7f880266a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:29:54.130 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.129+0000 7f880266a640 1 -- 172.21.15.17:0/783890441 --> v1:172.21.15.17:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f87fc0723c0 con 0x7f87fc14bba0 2024-03-29T11:29:54.130 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.129+0000 7f880266a640 1 librados: init done 2024-03-29T11:29:54.131 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.130+0000 7f880266a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:54.131 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.130+0000 7f880266a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:54.131 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.130+0000 7f880266a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:29:54.131 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.130+0000 7f880266a640 1 -- 172.21.15.17:0/783890441 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87fc070bf0 con 0x7f87fc14bba0 2024-03-29T11:29:54.131 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.130+0000 7f87f8ff9640 1 -- 172.21.15.17:0/783890441 <== mon.0 v1:172.21.15.17:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f87ec0020d0 con 0x7f87fc14bba0 2024-03-29T11:29:54.131 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.130+0000 7f87f8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:54.131 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.130+0000 7f87fa7fc640 4 set_mon_vals no callback set 2024-03-29T11:29:54.131 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.130+0000 7f87f8ff9640 1 -- 172.21.15.17:0/783890441 <== mon.0 v1:172.21.15.17:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3544029594 0 0) 0x7f87ec003b10 con 0x7f87fc14bba0 2024-03-29T11:29:54.131 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.130+0000 7f87f8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:54.131 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.130+0000 7f87f8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:54.131 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.130+0000 7f87f8ff9640 10 monclient: dump: 2024-03-29T11:29:54.131 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-03-29T11:29:54.132 INFO:teuthology.misc.health.smithi017.stderr:fsid a932e4f9-b3f4-4d39-b76e-25f5220acc19 2024-03-29T11:29:54.132 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:54.132 INFO:teuthology.misc.health.smithi017.stderr:created 2024-03-29T11:18:22.060865+0000 2024-03-29T11:29:54.132 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:54.132 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-03-29T11:29:54.132 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.a 2024-03-29T11:29:54.132 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.94:6789/0 mon.b 2024-03-29T11:29:54.132 INFO:teuthology.misc.health.smithi017.stderr:2: v1:172.21.15.17:6790/0 mon.c 2024-03-29T11:29:54.132 INFO:teuthology.misc.health.smithi017.stderr: 2024-03-29T11:29:54.132 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.132+0000 7f87f8ff9640 1 -- 172.21.15.17:0/783890441 <== mon.0 v1:172.21.15.17:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1641771999 0 0) 0x7f87ec019ae0 con 0x7f87fc14bba0 2024-03-29T11:29:54.132 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.132+0000 7f87f8ff9640 1 -- 172.21.15.17:0/783890441 <== mon.0 v1:172.21.15.17:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3682049282 0 0) 0x7f87ec0838f0 con 0x7f87fc14bba0 2024-03-29T11:29:54.132 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.132+0000 7f87f8ff9640 10 client.4578.objecter ms_dispatch 0x7f87fc000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:29:54.132 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.132+0000 7f87f8ff9640 3 client.4578.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:29:54.133 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.132+0000 7f87f8ff9640 3 client.4578.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:29:54.133 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.132+0000 7f87f8ff9640 20 client.4578.objecter dump_active .. 0 homeless 2024-03-29T11:29:54.133 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.133+0000 7f87fbfff640 10 monclient: get_auth_request con 0x7f87dc06de80 auth_method 0 2024-03-29T11:29:54.133 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.133+0000 7f87f8ff9640 10 client.4578.objecter ms_handle_connect 0x7f87dc06de80 2024-03-29T11:29:54.133 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.141+0000 7f87f8ff9640 1 -- 172.21.15.17:0/783890441 <== mon.0 v1:172.21.15.17:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f87ec04f230 con 0x7f87fc14bba0 2024-03-29T11:29:54.133 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.141+0000 7f87f8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:54.133 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.141+0000 7f87f8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:54.462 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.470+0000 7f880266a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:54.463 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.470+0000 7f880266a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:54.463 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.470+0000 7f880266a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.17:6789/0 2024-03-29T11:29:54.463 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.470+0000 7f880266a640 1 -- 172.21.15.17:0/783890441 --> v1:172.21.15.17:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f87fc13c940 con 0x7f87fc14bba0 2024-03-29T11:29:54.463 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.471+0000 7f87f8ff9640 1 -- 172.21.15.17:0/783890441 <== mon.0 v1:172.21.15.17:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 543280943) 0x7f87ec04f230 con 0x7f87fc14bba0 2024-03-29T11:29:54.463 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.471+0000 7f87f8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:54.463 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.471+0000 7f87f8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:54.464 INFO:teuthology.misc.health.smithi017.stdout: 2024-03-29T11:29:54.464 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":11,"fsid":"a932e4f9-b3f4-4d39-b76e-25f5220acc19","created":"2024-03-29T11:18:56.246535+0000","modified":"2024-03-29T11:29:01.208095+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:01.208095+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7583f99-c8fc-47b7-98c9-000b5892563e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"2b167bf1-6458-4b07-b603-ecceaf19933e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"b2fd6920-a919-4184-bce2-1aee9c71f90e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"cded4702-dbf5-48f9-9d2c-2b49173dee68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f43c11bd-586c-4992-bb35-04023637a513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27650969-0f63-4d75-b6c6-469e01ba4083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffc86661-1c24-44aa-a9aa-a45fe8816e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"379f15ab-e903-42a2-82cd-d222df0861e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:54.467 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.474+0000 7f880266a640 10 librados: watch_flush enter 2024-03-29T11:29:54.467 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.474+0000 7f880266a640 10 librados: watch_flush exit 2024-03-29T11:29:54.467 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.475+0000 7f880266a640 20 client.4578.objecter shutdown clearing up homeless session... 2024-03-29T11:29:54.467 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.475+0000 7f880266a640 10 client.4578.objecter successfully canceled tick 2024-03-29T11:29:54.467 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.475+0000 7f880266a640 1 -- 172.21.15.17:0/783890441 >> v1:172.21.15.17:6816/3236185052 conn(0x7f87dc06de80 legacy=0x7f87dc070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:54.467 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.475+0000 7f880266a640 10 monclient: shutdown 2024-03-29T11:29:54.467 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.475+0000 7f880266a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:54.467 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.475+0000 7f880266a640 1 -- 172.21.15.17:0/783890441 >> v1:172.21.15.17:6789/0 conn(0x7f87fc14bba0 legacy=0x7f87fc071970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:54.467 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.475+0000 7f880266a640 1 -- 172.21.15.17:0/783890441 shutdown_connections 2024-03-29T11:29:54.467 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.475+0000 7f880266a640 1 -- 172.21.15.17:0/783890441 >> 172.21.15.17:0/783890441 conn(0x7f87fc0b35e0 msgr2=0x7f87fc0aeff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:54.468 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.476+0000 7f880266a640 1 -- 172.21.15.17:0/783890441 shutdown_connections 2024-03-29T11:29:54.468 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.476+0000 7f880266a640 1 -- 172.21.15.17:0/783890441 wait complete. 2024-03-29T11:29:54.468 INFO:teuthology.misc.health.smithi017.stderr:2024-03-29T11:29:54.476+0000 7f880266a640 1 librados: shutdown 2024-03-29T11:29:54.488 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:54.488 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:29:54.490 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T11:29:54.490 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T11:29:54.490 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:29:54.532 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T11:29:54.532 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T11:29:54.532 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:29:54.558 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T11:29:54.559 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T11:29:54.559 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:29:54.629 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T11:29:54.630 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T11:29:54.630 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:29:54.702 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T11:29:54.703 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T11:29:54.703 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:29:54.739 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T11:29:54.739 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T11:29:54.740 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:29:54.814 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T11:29:54.814 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T11:29:54.814 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:29:54.883 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T11:29:54.883 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T11:29:54.883 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:29:54.962 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T11:29:54.962 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T11:29:54.963 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T11:29:54.963 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:29:54.987 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T11:29:54.988 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T11:29:54.988 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:29:55.027 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T11:29:55.027 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T11:29:55.027 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T11:29:55.027 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:29:55.039 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T11:29:55.039 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T11:29:55.039 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:29:55.090 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T11:29:55.090 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T11:29:55.090 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:29:55.102 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T11:29:55.103 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T11:29:55.103 DEBUG:teuthology.orchestra.run.smithi017:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T11:29:55.164 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi017.front.sepia.ceph.com 2024-03-29T11:29:55.164 DEBUG:teuthology.orchestra.run.smithi017:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T11:29:56.053 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi017.front.sepia.ceph.com 2024-03-29T11:29:56.054 DEBUG:teuthology.orchestra.run.smithi017:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T11:29:56.157 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi017.front.sepia.ceph.com 2024-03-29T11:29:56.157 DEBUG:teuthology.orchestra.run.smithi017:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T11:29:56.223 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi017.front.sepia.ceph.com 2024-03-29T11:29:56.224 DEBUG:teuthology.orchestra.run.smithi017:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T11:29:56.290 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi094.front.sepia.ceph.com 2024-03-29T11:29:56.290 DEBUG:teuthology.orchestra.run.smithi094:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T11:29:59.675 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi094.front.sepia.ceph.com 2024-03-29T11:29:59.675 DEBUG:teuthology.orchestra.run.smithi094:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T11:29:59.767 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi094.front.sepia.ceph.com 2024-03-29T11:29:59.767 DEBUG:teuthology.orchestra.run.smithi094:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T11:29:59.817 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi094.front.sepia.ceph.com 2024-03-29T11:29:59.817 DEBUG:teuthology.orchestra.run.smithi094:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T11:29:59.885 INFO:tasks.ceph:Archiving mon data... 2024-03-29T11:29:59.887 DEBUG:teuthology.misc:Transferring archived files from smithi017:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622683/data/mon.a.tgz 2024-03-29T11:29:59.887 DEBUG:teuthology.orchestra.run.smithi017:> mktemp 2024-03-29T11:29:59.905 INFO:teuthology.orchestra.run.smithi017.stdout:/tmp/tmp.b6WonH90tT 2024-03-29T11:29:59.906 DEBUG:teuthology.orchestra.run.smithi017:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.b6WonH90tT 2024-03-29T11:30:00.013 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0666 /tmp/tmp.b6WonH90tT 2024-03-29T11:30:00.102 DEBUG:teuthology.orchestra.remote:smithi017:/tmp/tmp.b6WonH90tT is 127KB 2024-03-29T11:30:00.154 DEBUG:teuthology.orchestra.run.smithi017:> rm -fr /tmp/tmp.b6WonH90tT 2024-03-29T11:30:00.170 DEBUG:teuthology.misc:Transferring archived files from smithi017:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622683/data/mon.c.tgz 2024-03-29T11:30:00.170 DEBUG:teuthology.orchestra.run.smithi017:> mktemp 2024-03-29T11:30:00.224 INFO:teuthology.orchestra.run.smithi017.stdout:/tmp/tmp.zexd6GCeMn 2024-03-29T11:30:00.224 DEBUG:teuthology.orchestra.run.smithi017:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.zexd6GCeMn 2024-03-29T11:30:00.314 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0666 /tmp/tmp.zexd6GCeMn 2024-03-29T11:30:00.398 DEBUG:teuthology.orchestra.remote:smithi017:/tmp/tmp.zexd6GCeMn is 127KB 2024-03-29T11:30:00.449 DEBUG:teuthology.orchestra.run.smithi017:> rm -fr /tmp/tmp.zexd6GCeMn 2024-03-29T11:30:00.463 DEBUG:teuthology.misc:Transferring archived files from smithi094:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622683/data/mon.b.tgz 2024-03-29T11:30:00.464 DEBUG:teuthology.orchestra.run.smithi094:> mktemp 2024-03-29T11:30:00.482 INFO:teuthology.orchestra.run.smithi094.stdout:/tmp/tmp.0YHjK4ca6I 2024-03-29T11:30:00.482 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.0YHjK4ca6I 2024-03-29T11:30:00.583 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0666 /tmp/tmp.0YHjK4ca6I 2024-03-29T11:30:00.671 DEBUG:teuthology.orchestra.remote:smithi094:/tmp/tmp.0YHjK4ca6I is 127KB 2024-03-29T11:30:00.722 DEBUG:teuthology.orchestra.run.smithi094:> rm -fr /tmp/tmp.0YHjK4ca6I 2024-03-29T11:30:00.737 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T11:30:00.737 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:30:00.740 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:30:00.819 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:30:00.861 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:30:00.861 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T11:30:00.863 DEBUG:teuthology.misc:Transferring archived files from smithi017:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622683/remote/smithi017/crash 2024-03-29T11:30:00.864 DEBUG:teuthology.orchestra.run.smithi017:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:30:00.890 DEBUG:teuthology.misc:Transferring archived files from smithi094:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622683/remote/smithi094/crash 2024-03-29T11:30:00.891 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:30:00.927 INFO:tasks.ceph:Compressing logs... 2024-03-29T11:30:00.928 DEBUG:teuthology.orchestra.run.smithi017:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:30:00.934 DEBUG:teuthology.orchestra.run.smithi094:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:30:01.000 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39171.log 2024-03-29T11:30:01.000 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40029.log 2024-03-29T11:30:01.001 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38742.log 2024-03-29T11:30:01.001 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.40029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39132.log 2024-03-29T11:30:01.001 INFO:teuthology.orchestra.run.smithi017.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39171.log.gz 2024-03-29T11:30:01.001 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.38742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38427.log 2024-03-29T11:30:01.001 INFO:teuthology.orchestra.run.smithi017.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40029.log.gz 2024-03-29T11:30:01.001 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T11:30:01.001 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39132.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38742.log.gz 2024-03-29T11:30:01.001 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.38427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41165.log 2024-03-29T11:30:01.001 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39873.log 2024-03-29T11:30:01.001 INFO:teuthology.orchestra.run.smithi017.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39132.log.gz 2024-03-29T11:30:01.001 INFO:teuthology.orchestra.run.smithi017.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38427.log.gz 2024-03-29T11:30:01.001 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39210.log 2024-03-29T11:30:01.001 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38781.log 2024-03-29T11:30:01.002 INFO:teuthology.orchestra.run.smithi017.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.41165.log.gz 2024-03-29T11:30:01.002 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40892.log 2024-03-29T11:30:01.002 INFO:teuthology.orchestra.run.smithi017.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39873.log.gz 2024-03-29T11:30:01.002 INFO:teuthology.orchestra.run.smithi017.stderr: 85.2% -- replaced with /var/log/ceph/ceph-client.admin.39210.log.gz 2024-03-29T11:30:01.002 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.38781.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28820.log 2024-03-29T11:30:01.002 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.40892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38898.log 2024-03-29T11:30:01.002 INFO:teuthology.orchestra.run.smithi017.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38781.log.gz 2024-03-29T11:30:01.002 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph.tmp-client.admin.28820.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28820.log.gz 2024-03-29T11:30:01.002 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38466.log 2024-03-29T11:30:01.002 INFO:teuthology.orchestra.run.smithi017.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40892.log.gz 2024-03-29T11:30:01.002 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.38898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39093.log 2024-03-29T11:30:01.002 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.38466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40307.log 2024-03-29T11:30:01.002 INFO:teuthology.orchestra.run.smithi017.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.38898.log.gz 2024-03-29T11:30:01.003 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39093.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.38466.log.gz 2024-03-29T11:30:01.003 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41243.log 2024-03-29T11:30:01.003 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.40307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39015.log 2024-03-29T11:30:01.003 INFO:teuthology.orchestra.run.smithi017.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39093.log.gz 2024-03-29T11:30:01.003 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41399.log 2024-03-29T11:30:01.003 INFO:teuthology.orchestra.run.smithi017.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40307.log.gz 2024-03-29T11:30:01.003 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39015.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41243.log.gz 2024-03-29T11:30:01.003 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41009.log 2024-03-29T11:30:01.003 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38859.log 2024-03-29T11:30:01.003 INFO:teuthology.orchestra.run.smithi017.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39015.log.gz 2024-03-29T11:30:01.003 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39288.log 2024-03-29T11:30:01.003 INFO:teuthology.orchestra.run.smithi017.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41399.log.gz 2024-03-29T11:30:01.003 INFO:teuthology.orchestra.run.smithi017.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41009.log.gz 2024-03-29T11:30:01.004 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38585.log 2024-03-29T11:30:01.004 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.38859.log: /var/log/ceph/ceph-client.admin.39288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40385.log 2024-03-29T11:30:01.004 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.38585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39561.log 2024-03-29T11:30:01.004 INFO:teuthology.orchestra.run.smithi017.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38859.log.gz 2024-03-29T11:30:01.004 INFO:teuthology.orchestra.run.smithi017.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39288.log.gz 2024-03-29T11:30:01.004 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.40385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41360.log 2024-03-29T11:30:01.004 INFO:teuthology.orchestra.run.smithi017.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38585.log.gz 2024-03-29T11:30:01.004 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40619.log 2024-03-29T11:30:01.004 INFO:teuthology.orchestra.run.smithi017.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40385.log.gz 2024-03-29T11:30:01.005 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41360.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39561.log.gz 2024-03-29T11:30:01.005 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40424.log 2024-03-29T11:30:01.005 INFO:teuthology.orchestra.run.smithi017.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41360.log.gz 2024-03-29T11:30:01.005 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40736.log 2024-03-29T11:30:01.005 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.40619.log: /var/log/ceph/ceph-client.admin.40424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39366.log 2024-03-29T11:30:01.005 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.40736.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40619.log.gz 2024-03-29T11:30:01.005 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40931.log 2024-03-29T11:30:01.005 INFO:teuthology.orchestra.run.smithi017.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40424.log.gz 2024-03-29T11:30:01.005 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40346.log 2024-03-29T11:30:01.005 INFO:teuthology.orchestra.run.smithi017.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40736.log.gz 2024-03-29T11:30:01.005 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.40931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41633.log 2024-03-29T11:30:01.005 INFO:teuthology.orchestra.run.smithi017.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39366.log.gz 2024-03-29T11:30:01.006 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.40346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40541.log 2024-03-29T11:30:01.006 INFO:teuthology.orchestra.run.smithi017.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40931.log.gz 2024-03-29T11:30:01.006 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40580.log 2024-03-29T11:30:01.006 INFO:teuthology.orchestra.run.smithi017.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40346.log.gz 2024-03-29T11:30:01.006 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.40541.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41633.log.gz 2024-03-29T11:30:01.006 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39249.log 2024-03-29T11:30:01.006 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.40580.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:30:01.006 INFO:teuthology.orchestra.run.smithi017.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40541.log.gz 2024-03-29T11:30:01.006 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39834.log 2024-03-29T11:30:01.006 INFO:teuthology.orchestra.run.smithi017.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.40580.log.gz 2024-03-29T11:30:01.006 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph.audit.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39249.log.gz 2024-03-29T11:30:01.007 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39600.log 2024-03-29T11:30:01.007 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39522.log 2024-03-29T11:30:01.007 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39600.log: gzip 85.3% -- replaced with /var/log/ceph/ceph-client.admin.39834.log.gz -5 2024-03-29T11:30:01.007 INFO:teuthology.orchestra.run.smithi017.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39717.log 2024-03-29T11:30:01.007 INFO:teuthology.orchestra.run.smithi017.stderr: 93.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:30:01.007 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41204.log 2024-03-29T11:30:01.007 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39522.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39600.log.gz 2024-03-29T11:30:01.007 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34918.log 2024-03-29T11:30:01.007 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38349.log 2024-03-29T11:30:01.007 INFO:teuthology.orchestra.run.smithi017.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39522.log.gz 2024-03-29T11:30:01.007 INFO:teuthology.orchestra.run.smithi017.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39717.log.gz 2024-03-29T11:30:01.008 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.34918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34627.log 2024-03-29T11:30:01.008 INFO:teuthology.orchestra.run.smithi017.stderr: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.41204.log.gz 2024-03-29T11:30:01.008 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.38349.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34918.log.gz 2024-03-29T11:30:01.008 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39990.log 2024-03-29T11:30:01.008 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.34627.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T11:30:01.008 INFO:teuthology.orchestra.run.smithi017.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38349.log.gz 2024-03-29T11:30:01.008 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38505.log 2024-03-29T11:30:01.008 INFO:teuthology.orchestra.run.smithi017.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.34627.log.gz 2024-03-29T11:30:01.008 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- 86.0% /var/log/ceph/ceph-client.admin.41087.log 2024-03-29T11:30:01.008 INFO:teuthology.orchestra.run.smithi017.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39990.log.gz 2024-03-29T11:30:01.008 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40970.log 2024-03-29T11:30:01.008 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.38505.log: /var/log/ceph/ceph-client.admin.41087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41048.log 2024-03-29T11:30:01.008 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.40970.log: gzip 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38505.log.gz 2024-03-29T11:30:01.008 INFO:teuthology.orchestra.run.smithi017.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39444.log 2024-03-29T11:30:01.008 INFO:teuthology.orchestra.run.smithi017.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41087.log.gz 2024-03-29T11:30:01.009 INFO:teuthology.orchestra.run.smithi017.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.40970.log.gz 2024-03-29T11:30:01.009 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38976.log 2024-03-29T11:30:01.009 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41711.log 2024-03-29T11:30:01.009 INFO:teuthology.orchestra.run.smithi017.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.41048.log.gz 2024-03-29T11:30:01.009 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.38976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38820.log 2024-03-29T11:30:01.009 INFO:teuthology.orchestra.run.smithi017.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39444.log.gz 2024-03-29T11:30:01.009 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40148.log 2024-03-29T11:30:01.009 INFO:teuthology.orchestra.run.smithi017.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38976.log.gz 2024-03-29T11:30:01.009 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.38820.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T11:30:01.009 INFO:teuthology.orchestra.run.smithi017.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.41711.log.gz 2024-03-29T11:30:01.010 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.40148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40265.log 2024-03-29T11:30:01.010 INFO:teuthology.orchestra.run.smithi017.stderr: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.38820.log.gz 2024-03-29T11:30:01.010 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38310.log 2024-03-29T11:30:01.010 INFO:teuthology.orchestra.run.smithi017.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40148.log.gz 2024-03-29T11:30:01.010 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39912.log 2024-03-29T11:30:01.010 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.40265.log: /var/log/ceph/ceph-client.admin.38310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39951.log 2024-03-29T11:30:01.010 INFO:teuthology.orchestra.run.smithi017.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40265.log.gz 2024-03-29T11:30:01.010 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39405.log 2024-03-29T11:30:01.010 INFO:teuthology.orchestra.run.smithi017.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38310.log.gz 2024-03-29T11:30:01.010 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40775.log 2024-03-29T11:30:01.010 INFO:teuthology.orchestra.run.smithi017.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39912.log.gz 2024-03-29T11:30:01.010 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40069.log 2024-03-29T11:30:01.011 INFO:teuthology.orchestra.run.smithi017.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39951.log.gz 2024-03-29T11:30:01.011 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.40775.log: gzip 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39405.log.gz 2024-03-29T11:30:01.011 INFO:teuthology.orchestra.run.smithi017.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40226.log 2024-03-29T11:30:01.011 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.40069.log: gzip -5 --verbose 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40775.log.gz 2024-03-29T11:30:01.011 INFO:teuthology.orchestra.run.smithi017.stderr: -- /var/log/ceph/ceph-client.admin.39756.log 2024-03-29T11:30:01.011 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.40226.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40069.log.gz 2024-03-29T11:30:01.011 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38544.log 2024-03-29T11:30:01.011 INFO:teuthology.orchestra.run.smithi017.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40226.log.gz 2024-03-29T11:30:01.011 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38624.log 2024-03-29T11:30:01.011 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41438.log 2024-03-29T11:30:01.011 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.38544.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39756.log.gz 2024-03-29T11:30:01.011 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.38624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39054.log 2024-03-29T11:30:01.011 INFO:teuthology.orchestra.run.smithi017.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.38544.log.gz 2024-03-29T11:30:01.011 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41438.log: gzip 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38624.log.gz 2024-03-29T11:30:01.012 INFO:teuthology.orchestra.run.smithi017.stderr: -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T11:30:01.012 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39054.log: gzip -5 --verbose 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41438.log.gz 2024-03-29T11:30:01.012 INFO:teuthology.orchestra.run.smithi017.stderr: -- /var/log/ceph/ceph-client.admin.41750.log 2024-03-29T11:30:01.012 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-mgr.y.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39054.log.gz 2024-03-29T11:30:01.012 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41477.log 2024-03-29T11:30:01.012 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40697.log 2024-03-29T11:30:01.012 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41126.log 2024-03-29T11:30:01.012 INFO:teuthology.orchestra.run.smithi017.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41750.log.gz 2024-03-29T11:30:01.012 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.40697.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41477.log.gz 2024-03-29T11:30:01.012 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35021.log 2024-03-29T11:30:01.012 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40187.log 2024-03-29T11:30:01.012 INFO:teuthology.orchestra.run.smithi017.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40697.log.gz 2024-03-29T11:30:01.012 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41126.log: /var/log/ceph/ceph-client.admin.35021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41555.log 2024-03-29T11:30:01.012 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.40187.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-29T11:30:01.012 INFO:teuthology.orchestra.run.smithi017.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35021.log.gz 2024-03-29T11:30:01.013 INFO:teuthology.orchestra.run.smithi017.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41126.log.gz 2024-03-29T11:30:01.013 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41555.log: gzip -5 87.5% --verbose -- /var/log/ceph/ceph-client.admin.41282.log 2024-03-29T11:30:01.013 INFO:teuthology.orchestra.run.smithi017.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40187.log.gz 2024-03-29T11:30:01.013 INFO:teuthology.orchestra.run.smithi017.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41555.log.gz/var/log/ceph/ceph-osd.0.log: 2024-03-29T11:30:01.013 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41594.log 2024-03-29T11:30:01.013 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39483.log 2024-03-29T11:30:01.013 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38703.log 2024-03-29T11:30:01.013 INFO:teuthology.orchestra.run.smithi017.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41282.log.gz 2024-03-29T11:30:01.013 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39483.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41594.log.gz 2024-03-29T11:30:01.013 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40502.log 2024-03-29T11:30:01.013 INFO:teuthology.orchestra.run.smithi017.stderr: 86.1%/var/log/ceph/ceph-client.admin.38703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41516.log 2024-03-29T11:30:01.013 INFO:teuthology.orchestra.run.smithi017.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39483.log.gz 2024-03-29T11:30:01.013 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.40502.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38703.log.gz 2024-03-29T11:30:01.013 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:30:01.014 INFO:teuthology.orchestra.run.smithi017.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40502.log.gz 2024-03-29T11:30:01.014 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T11:30:01.014 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41516.log: /var/log/ceph/ceph.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41516.log.gz 2024-03-29T11:30:01.014 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39678.log 2024-03-29T11:30:01.014 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose/var/log/ceph/ceph-osd.2.log: -- /var/log/ceph/ceph-client.admin.41321.log 2024-03-29T11:30:01.014 INFO:teuthology.orchestra.run.smithi017.stderr: 92.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:30:01.014 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38388.log 2024-03-29T11:30:01.014 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41321.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39678.log.gz 2024-03-29T11:30:01.014 INFO:teuthology.orchestra.run.smithi017.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41321.log.gz 2024-03-29T11:30:01.014 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41672.log 2024-03-29T11:30:01.014 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.38388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40814.log 2024-03-29T11:30:01.014 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41672.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38388.log.gz 2024-03-29T11:30:01.014 INFO:teuthology.orchestra.run.smithi017.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41672.log.gz 2024-03-29T11:30:01.014 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39639.log 2024-03-29T11:30:01.015 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.40814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39327.log 2024-03-29T11:30:01.015 INFO:teuthology.orchestra.run.smithi017.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40814.log.gz 2024-03-29T11:30:01.015 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38663.log 2024-03-29T11:30:01.015 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39327.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39639.log.gz 2024-03-29T11:30:01.015 INFO:teuthology.orchestra.run.smithi017.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39327.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38937.log 2024-03-29T11:30:01.015 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-03-29T11:30:01.015 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40463.log 2024-03-29T11:30:01.015 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.38663.log: /var/log/ceph/ceph-client.admin.38937.log: 86.1% 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38663.log.gz 2024-03-29T11:30:01.015 INFO:teuthology.orchestra.run.smithi017.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38937.log.gz 2024-03-29T11:30:01.015 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39795.log 2024-03-29T11:30:01.015 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40853.log 2024-03-29T11:30:01.015 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.40463.log: /var/log/ceph/ceph-client.admin.39795.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40463.log.gz 2024-03-29T11:30:01.015 INFO:teuthology.orchestra.run.smithi017.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39795.log.gz 2024-03-29T11:30:01.016 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40658.log 2024-03-29T11:30:01.016 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40109.log 2024-03-29T11:30:01.016 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.40853.log: /var/log/ceph/ceph-client.admin.40658.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40853.log.gz 2024-03-29T11:30:01.016 INFO:teuthology.orchestra.run.smithi017.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40658.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-29T11:30:01.016 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-03-29T11:30:01.016 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.40109.log: /var/log/ceph/ceph-mon.c.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40109.log.gz 2024-03-29T11:30:01.023 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T11:30:01.023 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34440.log 2024-03-29T11:30:01.023 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34240.log 2024-03-29T11:30:01.024 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.34440.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:30:01.024 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.34240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34320.log 2024-03-29T11:30:01.024 INFO:teuthology.orchestra.run.smithi094.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34440.log.gz 2024-03-29T11:30:01.024 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34200.log 2024-03-29T11:30:01.025 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph.audit.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34240.log.gz 2024-03-29T11:30:01.025 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.34320.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-29T11:30:01.025 INFO:teuthology.orchestra.run.smithi094.stderr: 88.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:30:01.025 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.34200.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T11:30:01.025 INFO:teuthology.orchestra.run.smithi094.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34320.log.gz 2024-03-29T11:30:01.025 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T11:30:01.025 INFO:teuthology.orchestra.run.smithi094.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34200.log.gz 2024-03-29T11:30:01.025 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34360.log 2024-03-29T11:30:01.025 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T11:30:01.026 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.34360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34280.log 2024-03-29T11:30:01.026 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- 87.8% /var/log/ceph/ceph.log 2024-03-29T11:30:01.026 INFO:teuthology.orchestra.run.smithi094.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34360.log.gz 2024-03-29T11:30:01.026 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.34280.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-29T11:30:01.026 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34280.log.gz 2024-03-29T11:30:01.026 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34400.log 2024-03-29T11:30:01.026 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34160.log 2024-03-29T11:30:01.026 INFO:teuthology.orchestra.run.smithi094.stderr: 87.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:30:01.026 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.34400.log: /var/log/ceph/ceph-mon.b.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34400.log.gz 2024-03-29T11:30:01.026 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.34160.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34160.log.gz 2024-03-29T11:30:01.027 INFO:teuthology.orchestra.run.smithi094.stderr: 93.9% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T11:30:01.053 INFO:teuthology.orchestra.run.smithi017.stderr: 92.1% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-29T11:30:12.704 INFO:teuthology.orchestra.run.smithi094.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T11:30:12.740 INFO:teuthology.orchestra.run.smithi094.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T11:30:12.759 INFO:teuthology.orchestra.run.smithi094.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T11:30:12.847 INFO:teuthology.orchestra.run.smithi017.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T11:30:13.001 INFO:teuthology.orchestra.run.smithi017.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T11:30:16.855 INFO:teuthology.orchestra.run.smithi094.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T11:30:17.103 INFO:teuthology.orchestra.run.smithi017.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T11:30:17.129 INFO:teuthology.orchestra.run.smithi017.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T11:30:51.400 INFO:teuthology.orchestra.run.smithi017.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T11:30:56.850 INFO:teuthology.orchestra.run.smithi017.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T11:30:56.852 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-03-29T11:30:56.852 INFO:teuthology.orchestra.run.smithi017.stderr:real 0m55.905s 2024-03-29T11:30:56.852 INFO:teuthology.orchestra.run.smithi017.stderr:user 2m37.073s 2024-03-29T11:30:56.852 INFO:teuthology.orchestra.run.smithi017.stderr:sys 0m5.174s 2024-03-29T11:31:10.298 INFO:teuthology.orchestra.run.smithi094.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T11:31:10.300 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T11:31:10.300 INFO:teuthology.orchestra.run.smithi094.stderr:real 1m9.308s 2024-03-29T11:31:10.300 INFO:teuthology.orchestra.run.smithi094.stderr:user 1m55.742s 2024-03-29T11:31:10.300 INFO:teuthology.orchestra.run.smithi094.stderr:sys 0m4.374s 2024-03-29T11:31:10.301 INFO:tasks.ceph:Archiving logs... 2024-03-29T11:31:10.301 DEBUG:teuthology.misc:Transferring archived files from smithi017:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622683/remote/smithi017/log 2024-03-29T11:31:10.302 DEBUG:teuthology.orchestra.run.smithi017:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:31:32.100 DEBUG:teuthology.misc:Transferring archived files from smithi094:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622683/remote/smithi094/log 2024-03-29T11:31:32.101 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:31:50.042 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:31:50.250 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=5a5727f78175418f8ea4cc7cf3e7d4a2 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:31:50.254 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T11:31:50.262 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T11:31:50.278 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T11:31:50.279 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:31:50.293 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:31:50.371 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T11:31:50.380 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T11:31:50.380 DEBUG:teuthology.orchestra.run.smithi017:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:31:50.383 DEBUG:teuthology.orchestra.run.smithi094:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:31:50.397 INFO:teuthology.orchestra.run.smithi017.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:31:50.398 INFO:teuthology.orchestra.run.smithi094.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:31:50.401 INFO:teuthology.orchestra.run.smithi017.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:31:50.401 INFO:teuthology.orchestra.run.smithi017.stdout:=============================================================================== 2024-03-29T11:31:50.402 INFO:teuthology.orchestra.run.smithi017.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 22 +470us[ +470us] +/- 91ms 2024-03-29T11:31:50.402 INFO:teuthology.orchestra.run.smithi017.stdout:^* hv02.front.sepia.ceph.com 3 6 377 23 -86us[ -87us] +/- 54ms 2024-03-29T11:31:50.402 INFO:teuthology.orchestra.run.smithi017.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 24 -267us[ -268us] +/- 86ms 2024-03-29T11:31:50.402 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:31:50.403 INFO:teuthology.orchestra.run.smithi094.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:31:50.404 INFO:teuthology.orchestra.run.smithi094.stdout:=============================================================================== 2024-03-29T11:31:50.404 INFO:teuthology.orchestra.run.smithi094.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 22 +487us[ +484us] +/- 91ms 2024-03-29T11:31:50.404 INFO:teuthology.orchestra.run.smithi094.stdout:^* hv02.front.sepia.ceph.com 3 6 377 21 -112us[ -116us] +/- 54ms 2024-03-29T11:31:50.404 INFO:teuthology.orchestra.run.smithi094.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 23 -267us[ -271us] +/- 86ms 2024-03-29T11:31:50.404 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:31:50.404 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T11:31:50.413 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T11:31:50.414 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T11:31:50.428 DEBUG:teuthology.orchestra.run.smithi017:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:31:50.514 DEBUG:teuthology.orchestra.run.smithi094:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:31:50.581 DEBUG:teuthology.orchestra.run.smithi017:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:31:50.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:31:50.613 DEBUG:teuthology.orchestra.run.smithi094:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:31:50.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:31:50.643 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T11:31:50.653 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T11:31:50.712 INFO:teuthology.task.internal:Duration was 1260.381971 seconds 2024-03-29T11:31:50.713 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T11:31:50.739 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T11:31:50.739 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:31:50.742 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:31:50.785 INFO:teuthology.orchestra.run.smithi017.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:31:50.786 INFO:teuthology.orchestra.run.smithi094.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:31:51.079 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T11:31:51.080 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi017.front.sepia.ceph.com 2024-03-29T11:31:51.080 DEBUG:teuthology.orchestra.run.smithi017:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:31:51.100 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi094.front.sepia.ceph.com 2024-03-29T11:31:51.100 DEBUG:teuthology.orchestra.run.smithi094:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:31:51.145 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T11:31:51.146 DEBUG:teuthology.orchestra.run.smithi017:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:31:51.148 DEBUG:teuthology.orchestra.run.smithi094:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:31:51.214 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T11:31:51.214 DEBUG:teuthology.orchestra.run.smithi017:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:31:51.217 DEBUG:teuthology.orchestra.run.smithi094:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:31:51.352 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T11:31:51.374 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T11:31:51.374 DEBUG:teuthology.orchestra.run.smithi017:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:31:51.401 DEBUG:teuthology.orchestra.run.smithi094:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:31:51.429 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T11:31:51.454 DEBUG:teuthology.orchestra.run.smithi017:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:31:51.458 DEBUG:teuthology.orchestra.run.smithi094:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:31:51.483 INFO:teuthology.orchestra.run.smithi017.stdout:kernel.core_pattern = core 2024-03-29T11:31:51.499 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern = core 2024-03-29T11:31:51.541 DEBUG:teuthology.orchestra.run.smithi017:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:31:51.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:31:51.564 DEBUG:teuthology.orchestra.run.smithi094:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:31:51.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:31:51.602 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T11:31:51.620 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T11:31:51.623 DEBUG:teuthology.misc:Transferring archived files from smithi017:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622683/remote/smithi017 2024-03-29T11:31:51.623 DEBUG:teuthology.orchestra.run.smithi017:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:31:51.674 DEBUG:teuthology.misc:Transferring archived files from smithi094:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622683/remote/smithi094 2024-03-29T11:31:51.675 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:31:51.724 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T11:31:51.724 DEBUG:teuthology.orchestra.run.smithi017:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:31:51.729 DEBUG:teuthology.orchestra.run.smithi094:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:31:51.766 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T11:31:51.779 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T11:31:51.780 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T11:31:51.810 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T11:31:51.811 DEBUG:teuthology.orchestra.run.smithi017:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:31:51.813 DEBUG:teuthology.orchestra.run.smithi094:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:31:51.824 INFO:teuthology.orchestra.run.smithi017.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:31 /home/ubuntu/cephtest 2024-03-29T11:31:51.827 INFO:teuthology.orchestra.run.smithi094.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:31 /home/ubuntu/cephtest 2024-03-29T11:31:51.828 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T11:31:51.847 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T11:31:51.881 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T11:31:51.903 DEBUG:teuthology.nuke:shortname: smithi017 2024-03-29T11:31:51.904 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:31:51.926 DEBUG:teuthology.nuke:shortname: smithi094 2024-03-29T11:31:51.926 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:31:51.930 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi017.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622683', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:02:04.965790', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGZcCOJjiFUA2a6jFFSoq93F3qAol6z1qPchs7Cw7fNOZHqYW7tnklnfaMxeM5zo+PIdJdeOTOw9XjhdIP5sXs0='} 2024-03-29T11:31:51.950 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi094.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622683', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:02:04.967455', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFplVfDj+ocYFtAOZXcnsLnocc/g2p4HL0RJGa2ZhOlaiDHJzgpAQBJR8Rqbd8sQotCrDibgV/AXAR1qM2gc4VI='} 2024-03-29T11:31:51.979 INFO:teuthology.orchestra.console.smithi017:Power off 2024-03-29T11:31:51.979 DEBUG:teuthology.orchestra.console.smithi017:pexpect command: ipmitool -H smithi017.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:31:52.007 DEBUG:teuthology.orchestra.console.smithi017:power off output: Chassis Power Control: Down/Off 2024-03-29T11:31:52.007 DEBUG:teuthology.orchestra.console.smithi017:pexpect command: ipmitool -H smithi017.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:31:52.027 INFO:teuthology.orchestra.console.smithi094:Power off 2024-03-29T11:31:52.028 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:31:52.038 DEBUG:teuthology.orchestra.console.smithi017:check power output: Chassis Power is on 2024-03-29T11:31:52.050 DEBUG:teuthology.orchestra.console.smithi094:power off output: Chassis Power Control: Down/Off 2024-03-29T11:31:52.051 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:31:52.077 DEBUG:teuthology.orchestra.console.smithi094:check power output: Chassis Power is on 2024-03-29T11:31:56.040 DEBUG:teuthology.orchestra.console.smithi017:pexpect command: ipmitool -H smithi017.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:31:56.078 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:31:56.154 DEBUG:teuthology.orchestra.console.smithi017:check power output: Chassis Power is on 2024-03-29T11:31:56.193 DEBUG:teuthology.orchestra.console.smithi094:check power output: Chassis Power is on 2024-03-29T11:32:00.162 DEBUG:teuthology.orchestra.console.smithi017:pexpect command: ipmitool -H smithi017.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:00.194 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:00.275 DEBUG:teuthology.orchestra.console.smithi017:check power output: Chassis Power is off 2024-03-29T11:32:00.306 DEBUG:teuthology.orchestra.console.smithi094:check power output: Chassis Power is off 2024-03-29T11:32:00.377 INFO:teuthology.orchestra.console.smithi017:Power off completed 2024-03-29T11:32:00.408 INFO:teuthology.orchestra.console.smithi094:Power off completed 2024-03-29T11:32:00.512 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} duration: 1260.3819711208344 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=5a5727f78175418f8ea4cc7cf3e7d4a2 status: fail success: false 2024-03-29T11:32:00.513 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:32:00.598 INFO:teuthology.run:FAIL